[OpenLayers-Users] a whole LOT of points!

[zach cruise]

we need to map points, a lot of them. they can only be vector and most
likely static geojson/svg - so no geoserver clustering in the backend
- also want to avoid maptiler:

1 mil records
200 mb geojson
download time 20 sec (over 1.5 mbps)
dom ready 15 sec (with 2 gb ram)

obviously we are also looking at modest maps in addition to openlayers.

any real world ideas?

can anyone point to a web map with the most vector points?

[Phil Scadden]

Post by zach cruise
any real world ideas?

In real world, processing a million points in a browser doesnt happen.
JS is slow, download time is slow, execution environment in a browser is
extremely limited. (in my experience, doing more than 1000 points of
geometry is unacceptably slow in single threaded). Easiest approach for
doing this in a browser begins with figuring how not to have vector
data. If this is an offline application, you want to look at local
server or local database. If you want to go extreme, then maybe look at
html5 web worker api to go multithreaded but I suspect you are up for
some pretty major rewrites of OL.

Notice: This email and any attachments are confidential.
If received in error please destroy and immediately notify us.
Do not copy or disclose the contents.

[Stephen Woodbridge]

Post by Phil Scadden

Post by zach cruise
any real world ideas?

In real world, processing a million points in a browser doesnt happen.
JS is slow, download time is slow, execution environment in a browser is
extremely limited. (in my experience, doing more than 1000 points of
geometry is unacceptably slow in single threaded). Easiest approach for
doing this in a browser begins with figuring how not to have vector
data. If this is an offline application, you want to look at local
server or local database. If you want to go extreme, then maybe look at
html5 web worker api to go multithreaded but I suspect you are up for
some pretty major rewrites of OL.

I would concur with this.
What is the basis for the requirement to do all this in the browser?
Is it an online or offline application?
If offline is there any reason you can not have a local database and
server on the offline machine?

What is the user going to be trying to do with all these point?

If you have a million points, they can't all be displayed on the screen
at the same time or I would expect that they would all overlap to the
point that you would be hard pressed to display them all in a useful
manner where the user could differentiate them.

If you are displaying a subset then why not load just the ones you need?
using something like the bounding box strategy.

I think you need to provide more information about what and how you plan
to use this before people can give you better advice.

-Steve w

[Max Demars]

I imagine a sort of asynchronous streaming of json (maybe vectiles) in
combination of clustering strategies could work,

However I cannot give any example of this, but I dont think that would be
that hard to develop.

--Max Demars

Post by Stephen Woodbridge

Post by Phil Scadden

Post by zach cruise
any real world ideas?

In real world, processing a million points in a browser doesnt happen.
JS is slow, download time is slow, execution environment in a browser is
extremely limited. (in my experience, doing more than 1000 points of
geometry is unacceptably slow in single threaded). Easiest approach for
doing this in a browser begins with figuring how not to have vector
data. If this is an offline application, you want to look at local
server or local database. If you want to go extreme, then maybe look at
html5 web worker api to go multithreaded but I suspect you are up for
some pretty major rewrites of OL.

I would concur with this.
What is the basis for the requirement to do all this in the browser?
Is it an online or offline application?
If offline is there any reason you can not have a local database and
server on the offline machine?
What is the user going to be trying to do with all these point?
If you have a million points, they can't all be displayed on the screen at
the same time or I would expect that they would all overlap to the point
that you would be hard pressed to display them all in a useful manner where
the user could differentiate them.
If you are displaying a subset then why not load just the ones you need?
using something like the bounding box strategy.
I think you need to provide more information about what and how you plan
to use this before people can give you better advice.
-Steve w
_______________________________________________
Users mailing list
http://lists.osgeo.org/mailman/listinfo/openlayers-users

[Umberto Nicoletti]

Here's a link to my response to a similar question on gis.se:

http://gis.stackexchange.com/questions/18572/openlayers-alternatives-supporting-more-client-side-features/18591#18591

While I am not sure you can push 1M records to the browser without blowing
it up it gives some pointers to get started with.

hth,
Umberto

Post by zach cruise
we need to map points, a lot of them. they can only be vector and most
likely static geojson/svg - so no geoserver clustering in the backend
1 mil records
200 mb geojson
download time 20 sec (over 1.5 mbps)
dom ready 15 sec (with 2 gb ram)
obviously we are also looking at modest maps in addition to openlayers.
any real world ideas?
can anyone point to a web map with the most vector points?
_______________________________________________
Users mailing list
http://lists.osgeo.org/mailman/listinfo/openlayers-users

[Bart van den Eijnden]

Also make sure to checkout the WebGL renderer in OpenLayers 3.

Best regards,
Bart

Bart van den Eijnden
Front End Software Engineer | Boundless <http://boundlessgeo.com/>
***@boundlessgeo.com <mailto:***@boundlessgeo.com>
1-877-673-6436 <tel:[phone]>
@boundlessgeo <http://twitter.com/boundlessgeo/>

<http://boundlessgeo.com/>

http://gis.stackexchange.com/questions/18572/openlayers-alternatives-supporting-more-client-side-features/18591#18591 <http://gis.stackexchange.com/questions/18572/openlayers-alternatives-supporting-more-client-side-features/18591#18591>
While I am not sure you can push 1M records to the browser without blowing it up it gives some pointers to get started with.
hth,
Umberto
we need to map points, a lot of them. they can only be vector and most
likely static geojson/svg - so no geoserver clustering in the backend
1 mil records
200 mb geojson
download time 20 sec (over 1.5 mbps)
dom ready 15 sec (with 2 gb ram)
obviously we are also looking at modest maps in addition to openlayers.
any real world ideas?
can anyone point to a web map with the most vector points?
_______________________________________________
Users mailing list
http://lists.osgeo.org/mailman/listinfo/openlayers-users <http://lists.osgeo.org/mailman/listinfo/openlayers-users>
_______________________________________________
Users mailing list
http://lists.osgeo.org/mailman/listinfo/openlayers-users

[mortac8]

Use the OL3/Cesium(WebGL) stuff like Bart said. Using the Cesium 3D engine,
you can display 5-10million+ points using the Cesium pointGeometry branch
(which is on github).



--
View this message in context: http://osgeo-org.1560.x6.nabble.com/a-whole-LOT-of-points-tp5183473p5183750.html
Sent from the OpenLayers Users mailing list archive at Nabble.com.

[Phil Scadden]

Post by mortac8
Use the OL3/Cesium(WebGL) stuff like Bart said. Using the Cesium 3D engine,
you can display 5-10million+ points using the Cesium pointGeometry branch

Count me skeptical, but I can see the WebGL might render point much
faster than drawing on a canvas - once you have got the points into
memory in the first place. Downloading as json and then interpreting
this into memory for 1M points is going to be slow. I note big data
going to cesium like terrain server support or 3D models are in packed
binary formats instead with json shell and are passed directly to host
graphics subsystem. If you can organize your data like that, you are
good to go but I am not sure you can then leverage advantages of vector
formats in openlayers.

Notice: This email and any attachments are confidential.
If received in error please destroy and immediately notify us.
Do not copy or disclose the contents.

[zach cruise]

I dont know if you are using Openlayers 2 or 3 but both has points
see: http://openlayers.org/en/v3.0.0/examples/cluster.html

from http://openlayers.org/en/v3.0.0/examples/cluster.js, it is
randomly creating points. what if it had to load points already
created in memory? can ol2 (or 3) similarly cluster a 200 mb geojson
file?

or this: http://stackoverflow.com/a/6642203

Post by zach cruise
any real world ideas?

In real world, processing a million points in a browser doesnt happen.
JS is slow, download time is slow, execution environment in a browser is
extremely limited. (in my experience, doing more than 1000 points of
geometry is unacceptably slow in single threaded). Easiest approach for
doing this in a browser begins with figuring how not to have vector
data.

i want to avoid maptiler but what are the options for creating custom
tiles in ol?

Post by zach cruise

Post by zach cruise
any real world ideas?

In real world, processing a million points in a browser doesnt happen.
JS is slow, download time is slow, execution environment in a browser is
extremely limited. (in my experience, doing more than 1000 points of
geometry is unacceptably slow in single threaded). Easiest approach for
doing this in a browser begins with figuring how not to have vector
data. If this is an offline application, you want to look at local
server or local database. If you want to go extreme, then maybe look at
html5 web worker api to go multithreaded but I suspect you are up for
some pretty major rewrites of OL.

I would concur with this.
What is the basis for the requirement to do all this in the browser?
Is it an online or offline application?

online

If you have a million points, they can't all be displayed on the screen
at the same time or I would expect that they would all overlap to the
point that you would be hard pressed to display them all in a useful
manner where the user could differentiate them.
If you are displaying a subset then why not load just the ones you need?
using something like the bounding box strategy.

agree. but how does clustering happen with a 200 mb geojson file?
doesn't it need to load everything in memory first? if it is chunked,
how does ol know what chunk to call next (or asynchronous streaming of
vectiles as max said.
http://mattmakesmaps.com/blog/2013/10/09/tilestache-rendering-topojson/
uses a postgis backend that i can't use)?

http://gis.stackexchange.com/questions/18572/openlayers-alternatives-supporting-more-client-side-features/18591#18591

this talks about a timer and filtering the visible points server-side
by configuring a strategy option to the vector layer. i can't filter
server-side, just have a 200 mb geojson to map.

Post by zach cruise
Use the OL3/Cesium(WebGL) stuff like Bart said. Using the Cesium 3D engine,
you can display 5-10million+ points using the Cesium pointGeometry branch

Count me skeptical, but I can see the WebGL might render point much
faster than drawing on a canvas - once you have got the points into
memory in the first place. Downloading as json and then interpreting
this into memory for 1M points is going to be slow. I note big data
going to cesium like terrain server support or 3D models are in packed
binary formats instead with json shell and are passed directly to host
graphics subsystem. If you can organize your data like that, you are
good to go but I am not sure you can then leverage advantages of vector
formats in openlayers.

agree with phil. dom ready takes 15 sec.

[Stephen Woodbridge]

Post by zach cruise

I dont know if you are using Openlayers 2 or 3 but both has points
see: http://openlayers.org/en/v3.0.0/examples/cluster.html

from http://openlayers.org/en/v3.0.0/examples/cluster.js, it is
randomly creating points. what if it had to load points already
created in memory? can ol2 (or 3) similarly cluster a 200 mb geojson
file?

or this: http://stackoverflow.com/a/6642203

Post by zach cruise
any real world ideas?

In real world, processing a million points in a browser doesnt happen.
JS is slow, download time is slow, execution environment in a browser is
extremely limited. (in my experience, doing more than 1000 points of
geometry is unacceptably slow in single threaded). Easiest approach for
doing this in a browser begins with figuring how not to have vector
data.

i want to avoid maptiler but what are the options for creating custom
tiles in ol?

Post by zach cruise

Post by zach cruise
any real world ideas?

In real world, processing a million points in a browser doesnt happen.
JS is slow, download time is slow, execution environment in a browser is
extremely limited. (in my experience, doing more than 1000 points of
geometry is unacceptably slow in single threaded). Easiest approach for
doing this in a browser begins with figuring how not to have vector
data. If this is an offline application, you want to look at local
server or local database. If you want to go extreme, then maybe look at
html5 web worker api to go multithreaded but I suspect you are up for
some pretty major rewrites of OL.

I would concur with this.
What is the basis for the requirement to do all this in the browser?
Is it an online or offline application?

online

If you have a million points, they can't all be displayed on the screen
at the same time or I would expect that they would all overlap to the
point that you would be hard pressed to display them all in a useful
manner where the user could differentiate them.
If you are displaying a subset then why not load just the ones you need?
using something like the bounding box strategy.

agree. but how does clustering happen with a 200 mb geojson file?
doesn't it need to load everything in memory first? if it is chunked,
how does ol know what chunk to call next (or asynchronous streaming of
vectiles as max said.
http://mattmakesmaps.com/blog/2013/10/09/tilestache-rendering-topojson/
uses a postgis backend that i can't use)?

Ok the idea behind vector tiles is that you do not load the whole 200mb
geojson file, you break it into smaller regular tiles like google raster
tiles for maps. Each of these has a name that maps to its extents and
these are loaded as needed based on your view extents. and they can
cached by the browser if they are static.

And for zoom levels you might create tiles, that are pre-clustered, so
as you zoom out the tiles are already in a clustered geojson format, so
you only display the clustered points, but each of these has all the
underlying data or statistics that you need. This obvious assume that
you have some what to tile your 200md geojson file into smaller regular
tiles, and that you have some way to pre cluster you tiles if you want that.

-Steve

Post by zach cruise

http://gis.stackexchange.com/questions/18572/openlayers-alternatives-supporting-more-client-side-features/18591#18591

this talks about a timer and filtering the visible points server-side
by configuring a strategy option to the vector layer. i can't filter
server-side, just have a 200 mb geojson to map.

Post by zach cruise
Use the OL3/Cesium(WebGL) stuff like Bart said. Using the Cesium 3D engine,
you can display 5-10million+ points using the Cesium pointGeometry branch

Count me skeptical, but I can see the WebGL might render point much
faster than drawing on a canvas - once you have got the points into
memory in the first place. Downloading as json and then interpreting
this into memory for 1M points is going to be slow. I note big data
going to cesium like terrain server support or 3D models are in packed
binary formats instead with json shell and are passed directly to host
graphics subsystem. If you can organize your data like that, you are
good to go but I am not sure you can then leverage advantages of vector
formats in openlayers.

agree with phil. dom ready takes 15 sec.
_______________________________________________
Users mailing list
http://lists.osgeo.org/mailman/listinfo/openlayers-users

[Phil Scadden]

can ol2 (or 3) similarly cluster a 200 mb geojson file?

very slowly.

i want to avoid maptiler but what are the options for creating custom
tiles in ol?

ol consumes tiles served by a map tiling service. It doesnt create them.
The value of tile servers is that they move the rendering and querying
of spatial databases off the client which is slow at both (can only
query a spatial database by reading the whole thing into memory). I dont
know your objection to maptiler or geoserver or any of the other tiling
servers around, but if you create a custom service, you can fetch them
into OL by overriding getURL method.

agree. but how does clustering happen with a 200 mb geojson file?

doesn't it need to load everything in memory first? if it is chunked,

how does ol know what chunk to call next (or asynchronous streaming of

vectiles as max said.
http://mattmakesmaps.com/blog/2013/10/09/tilestache-rendering-topojson/
uses a postgis backend that i can't use)?

If the application is online, then I cant really understand why you dont
use a server. Clustering can happen serverside too and then you dont
have any 200mb files on the client. I am frankly of the opinion that you
cant do 1m point mapping without using serverside processing.

this talks about a timer and filtering the visible points server-side

by configuring a strategy option to the vector layer. i can't filter

server-side, just have a 200 mb geojson to map.

Then I think you are stuffed. You need to rethink to get the geojson
managed serverside - preferably into a spatial database. If the
application is online, then you have a server. Use it.


Notice: This email and any attachments are confidential.
If received in error please destroy and immediately notify us.
Do not copy or disclose the contents.

[Harris Hudson]

Due to network download speeds/data volume and browser vector
rendering limitations, I resorted to a compromise by using
BBOX strategy and server side clustering/filtering in an OL2
application I did a while ago to handle a layer of more than
70,000 points (but with still some limitations for older browsers).

The current application response time is a little poor but that
is due simply to old server hardware it is currently running
on (once the backend is upgraded in a few months time, most map
requests will be sub-second) - from the front end, using a
compromise of server side clustering/filtering, the OL2 side
of things can give the illusion of handling a huge number of
points.

http://web.acma.gov.au/pls/radcom/site_proximity.main_page

[Martin Davis]

I also ended up using server-side rendering in an application for
displaying crime incident points. Doing the map rendering server-side also
allowed nice features such as enforcing access security and rich symbology.

We wrote our own very simple map service, rendering points from a database
query onto an image.

We didn't even use a spatial database - for X,Y data simply using standard
B-tree indices was plenty fast enough.

Post by Harris Hudson
Due to network download speeds/data volume and browser vector
rendering limitations, I resorted to a compromise by using
BBOX strategy and server side clustering/filtering in an OL2
application I did a while ago to handle a layer of more than
70,000 points (but with still some limitations for older browsers).
The current application response time is a little poor but that
is due simply to old server hardware it is currently running
on (once the backend is upgraded in a few months time, most map
requests will be sub-second) - from the front end, using a
compromise of server side clustering/filtering, the OL2 side
of things can give the illusion of handling a huge number of
points.
http://web.acma.gov.au/pls/radcom/site_proximity.main_page
_______________________________________________
Users mailing list
http://lists.osgeo.org/mailman/listinfo/openlayers-users

[mortac8]

We have drag&dropped and parsed a 330MB .las file (10million points) and
displayed 2million de-densified points in about 30seconds (includes file
"uploading", parsing and initial drawing time) on Cesium in Chrome running
at ~15fps on a 3 year old laptop.

Parsing GeoJSON is a speed roadblock but your task can be done at the speeds
you want all in the browser.



--
View this message in context: http://osgeo-org.1560.x6.nabble.com/a-whole-LOT-of-points-tp5183473p5184019.html
Sent from the OpenLayers Users mailing list archive at Nabble.com.

[mortac8]

See an example here http://plas.io/
That is not my project but it was a major inspiration for it.



--
View this message in context: http://osgeo-org.1560.x6.nabble.com/a-whole-LOT-of-points-tp5183473p5184020.html
Sent from the OpenLayers Users mailing list archive at Nabble.com.

[Dalton Filho]

I don't know if this applies to your situation:

Some real state websites need to display several properties in a limited
space. What they do is to use aggregation circles that display how many
points are aggregated in that area. Once the user clicks the circle, the
map zooms in revealing yet more aggregation circles. The process repeats
until an acceptable number of properties can be displayed.

http://www.mls.ca
http://www.buzzbuzzhome.com

Post by mortac8
See an example here http://plas.io/
That is not my project but it was a major inspiration for it.
--
http://osgeo-org.1560.x6.nabble.com/a-whole-LOT-of-points-tp5183473p5184020.html
Sent from the OpenLayers Users mailing list archive at Nabble.com.
_______________________________________________
Users mailing list
http://lists.osgeo.org/mailman/listinfo/openlayers-users

[Phil Scadden]

Post by mortac8
We have drag&dropped and parsed a 330MB .las file (10million points) and
displayed 2million de-densified points in about 30seconds (includes file
"uploading", parsing and initial drawing time) on Cesium in Chrome running
at ~15fps on a 3 year old laptop.

I would note that .las is binary format. How much processing of those
points was by browser js? (as opposed to offloading to opengl). There is
no way to get that kind of throughput from geoJSON?