Home Wiki Blog Forum GEXF.net

Gephi forums

Community support

another CSV spreadsheet/memory issue.

Once it's running

another CSV spreadsheet/memory issue.

Postby lucsampaio » 09 Jan 2014 15:43

Hi there. I'm just a beginner in Gephi, so bear with me if it's been answered before (I tried the search, and did not get - as far as I could tell - a hit with enough similarity to my issue.)

I currently am trying to analyze an undirected network consisting of 6k nodes and some 18m edges (yes, lots and lots of interconnection to so few nodes), as a way to identify the most significant shared posts accross a number of fanpages in facebook. The data was extracted through netvizz, ranked first by type (I'm only interested in photos) and then share count. that gave me the 6k+ images I'm analyzing. afterwards I categorized these images through several attributes.

Each image is a node, and the edges are shared attributes between them (thus the high edge count)

The node spreadsheet folows this setup:

ID,Label,Texto,categoria,Imagem,Descricao,Call,Color
00001,00001,não,foto,cotidiano,sociedade,não,#FF0000
00002,00002,sim,foto,retrato,relacionamentos,sim,#FF0000
00003,00003,sim,ilustracao,retrato,religiao,sim,#00FF00
00004,00004,sim,foto,midia,comparacao,sim,#FF0000
00005,00005,não,foto,cotidiano,sociedade,não,#FF0000
00006,00006,sim,foto,objetos,nostalgia,não,#FF0000
00007,00007,não,foto,cotidiano,sentimentos,não,#FF0000
00008,00008,sim,foto,retrato,estiloVida,não,#FF0000
00009,00009,sim,manipulacao,ambiente,religiao,sim,#0000FF
00010,00010,não,foto,midia,relacionamentos,sim,#FF0000
[...]

while the edges spreadsheets - I have two types of edge spreadsheets: one correlating all attributes, and several others counting only pairs of attributes.I plan on building several graphs to visualize different correlations in order to find proper significance of the imagery - (processed from the nodes, to count same attributes and give this as weight) follow this structure:

Source,Target,Type,Texto,Categoria,Imagem,Descricao,CallToAction,Weight
00001,00002,undirected,não,sim,não,não,não,1
00001,00004,undirected,não,sim,não,não,não,1
00001,00005,undirected,sim,sim,sim,sim,sim,5
00001,00006,undirected,não,sim,não,não,sim,2
00001,00007,undirected,sim,sim,sim,não,sim,4
00001,00008,undirected,não,sim,não,não,sim,2
00001,00010,undirected,sim,sim,não,não,não,2
00001,00011,undirected,sim,sim,não,não,sim,3
00001,00013,undirected,sim,sim,não,não,sim,3
00001,00014,undirected,não,sim,não,não,não,1
[...]

(each "sim" marks a same-value attribute between nodes and is counted as a 1 and added as edge weight during parsing of the node file)

My problem exactly:

I can get the nodes into data laboratory without any issues, but the edges file hanged in all attempts I've made. At first I tried in 32-bit Debian (1500m memory limit), then in 64-bit Mac OS X (2g mem limt) , and finally in 64-bit Windows 8 (15g mem limit).

Since none of these computers really crashed (just gephi got hanged) during processing, I'm wondering if it might be an issue with my edge spreadsheet that makes it impossible to properly import this file. If such is the case, any idea on how could I get it to work?

thanks a lot in advance for any help that comes my way.
lucsampaio
 
Posts: 2
Joined: 09 Jan 2014 14:59
Location: Curitiba - Parana - Brazil

Re: another CSV spreadsheet/memory issue.

Postby pegerp » 09 Jan 2014 16:05

Hello lucsampaio.

I've been working with similar large graphs and memory consumption is a bit of a problem. See posts at http://forum.gephi.org/viewtopic.php?p=7941#p7941 and viewtopic.php?p=7826#p7826

Have you tried with smaller amount of edges? Ie. try to load only 1M edges and see what happens. If adding edges hangs the process you know that it is memory related. In the second link you can see that I used 40G -Xmx setting although my machine only has 24G installed. This way I got Gephi to load the whole network I was working on.

Good luck!
pegerp
 
Posts: 101
Joined: 21 Dec 2011 18:10

Re: another CSV spreadsheet/memory issue.

Postby lucsampaio » 09 Jan 2014 16:45

thanks pegerp!

I'll check the topics right away, and try the less edges approach. as soon as i get it, I'll get back to you. :)
lucsampaio
 
Posts: 2
Joined: 09 Jan 2014 14:59
Location: Curitiba - Parana - Brazil


Return to How-To and Troubleshooting

Who is online

Users browsing this forum: No registered users and 1 guest