Home Wiki Blog Forum GEXF.net

Gephi forums

Community support

openord and graph 4M nodes, 100M weighted arcs

Connection with databases, scaling to million-node networks

openord and graph 4M nodes, 100M weighted arcs

Postby gg4gephi » 18 Sep 2013 23:57

Hi!

I've been using gephi for simple graphs.
I was used to use Force Directed for layout.

Now I need to represent networks of 4M nodes and 100M weighted arcs.
I'd like to experiment with OpenOrd layout. Will it handle it?

I need to choose the proper file format and understand how to set parameters properly.

I think the easiest and more compact would be a CSV with adjacency list and weights.

I tried the format

a;b;0.12112
b;c;0.43;d;0.8

but values are understood as labels (nodes).

How to feed data in form of adjaceny matrix in a compact way?
I'd like also to use labels and filter them based on weighted edges or other properties: would you suggest gdf or which other way?

Last question: any javascript project using openOrd on web or visualizing such a massive graph on web interactively?
gg4gephi
 
Posts: 4
Joined: 18 Sep 2013 23:48
Location: Italy

Re: openord and graph 4M nodes, 100M weighted arcs

Postby pegerp » 19 Sep 2013 11:32

Hi gg4gephi

In my own experience the OpenOrd algorithm did not produce as good layout as I'd liked it to. I tested OpenOrd on a network that had 1-10M nodes and maybe 5-50M edges depending on what filtering was done beforehand. Later I started to use Force Atlas 2 which produces lots of better layout and can handle node sizes (no overlapping).

I've used both VNA and GDF formats. GDF is probably better although the file formats are quite similar. GDF is easy to open with File > Open... in Gephi. CSV files need to be imported which I think is more tedious. Generating GDFs programmatically is also quite easy because edges and nodes are defined on single lines.

For visualizing large networks there are some tools but none to my knowledge that are particularly finished. Seadragon plugin is in Gephi. It renders a large picture and chops it into smaller pieces. Not suitable for extremely large networks as yours.

SigmaJS is a JavaScript library that is very good for visualizing networks that are smaller than few thousand nodes.

I've been doing my own visualization for large network myself but the project is still work-in-progress. It is not likely to be complete plugin or anything like that in near future either. However it's possible to render large networks as long as the network fits in the memory. Currently rendering times are also quite slow due to missing optimizations. You can see an example network at http://85.25.226.110/mapper. The data is in MySQL database so the frontend JavaScript can request data that is currently visible on the screen. This way the backend can hold gigabytes of data that the frontend only loads on-demand.

Spinoff of that project is similar visualization which loads all the node data in a JSON at page load, http://85.25.226.110/mapper/map/pege. It'd be great to make a generic plugin out of these projects but currently my time is limited for these. Check this Gephi thread for more info, http://forum.gephi.org/viewtopic.php?f=28&t=2314

Another Gepher, solipsy, posted about how to publish large networks on MapBox: http://forum.gephi.org/viewtopic.php?f=28&t=2518&p=8306. I talked with him over PM on these forums and he seemed interested in helping others with this stuff. I'd be happy to give more information if it's needed. I hope someone can use this procedure to publish their own map. You may want to send him PM so he gets an email alert about new message at Gephi forums.

Good luck! Let us know if you get any results.
pegerp
 
Posts: 101
Joined: 21 Dec 2011 18:10

Re: openord and graph 4M nodes, 100M weighted arcs

Postby gg4gephi » 20 Sep 2013 00:26

Hi Pegerp!
thank you for your answer!

I looked at your links and it sounds pretty interesting what you did, especially for the possibility to load the network "on demand" with the GBytes on the server.

However, I am looking for a simple solution I can handle - I am not an expert of gephi and scripting, I've just used it.
I also played with Javascript libraries, maybe webgl can handle K of nodes, but not millions.

I would be happy to contact solipsy, cause he's using external services (mapbox) to visualize the network, it may be useful.
I ll send him a PM too.
I'd be happy even if I can just create a picture of it, no need necessarily to be interactive.

With gephi so far I manipulated netwrok of 16K nodes and 400K edges, but it is sweating....

Maybe sm other possiblity to handle the graph and get a picture? smtg like the front-end cover of "network, and introduction" book it owuld be cool!

Last question:
I feel cousy in using weighted adjacency lists.

I see csv is farily easy in getting the adj list, not weighted.

I also created a script to generate the gdf files, it is ok, but for huge network I am afraid it'd be much more redundant than a simple csv.

Any way in using a weighted adj list in a compact way?

thank you again pegerp!
gg4gephi
 
Posts: 4
Joined: 18 Sep 2013 23:48
Location: Italy

Re: openord and graph 4M nodes, 100M weighted arcs

Postby gg4gephi » 20 Sep 2013 00:35

Actually I have been amazed by your visualizations:
I'd like to learn how to place them on Gmap too!
I see you handle 100M arcs and 3M nodes too, so I think your help will be very precious!

Could we arrange a skype for guideline me how to do it?

basically, I am always working with weighted adj list : I have them in txt - around 2GB.
thank you again and my comngratualations also for the choices of colors!
http://85.25.226.110/mapper/plrnet-1M-10M-color?noidle

thank you again!
gg4gephi
 
Posts: 4
Joined: 18 Sep 2013 23:48
Location: Italy


Return to Databases & Very Large Networks

Who is online

Users browsing this forum: No registered users and 1 guest