Compressed graph

A compact and highly-efficient representation of the graph dataset, suited for scale-up analysis on high-end machines with large amounts of memory. The graph is compressed in Boldi-Vigna representation, designed to be loaded by the WebGraph framework, specifically using our swh-graph library.

Comments: This graph changed the MPH from GOV/Cmph to PTHash; Rust code hardcoding GOVMPH needs to replace it with DynMph or SwhidPthash. Java is no longer supported to read this graph.
Dataset size: 11 TiB
Export date: 2024-08-23
Teaser dataset: Popular 500 python compressed graph (15 GB); Popular 500 python provenance (all releases and revisions) (46 GB)
S3 URL: s3://softwareheritage/graph/2024-08-23/compressed/
Deprecated: False

Referencing Software Heritage

If you use any of the datasets indexed on this website for research purposes, please acknowledge Software Heritage as recommended in the publications page, that is:

Add a footnote on the title page of your paper, formatted as: “This work was made possible by Software Heritage, the universal source code archive: https://www.softwareheritage.org”; and
cite at least one of the following papers:
- Roberto Di Cosmo, Stefano Zacchiroli. Software Heritage: Why and How to Preserve Software Source Code. iPRES 2017. (BibTeX)
- Jean-François Abramatic, Roberto Di Cosmo, Stefano Zacchiroli. Building the universal archive of source code. Commun. ACM 61(10): 29-31 (2018). (BibTeX)

Specific datasets might recommend additional citations, to credit their creators.

Specific citation instructions

If you use this dataset for research purposes, please cite the following paper: Antoine Pietri, Diomidis Spinellis, Stefano Zacchiroli. The Software Heritage Graph Dataset: Public software development under one roof. In proceedings of MSR 2019: The 16th International Conference on Mining Software Repositories, May 2019, Montreal, Canada. Co-located with ICSE 2019 (Preprint), (BibTeX).

Download the dataset

For Amazon S3 links, you'll need to install either awscli or swh.datasets.

aws s3 cp --recursive --no-sign-request s3://softwareheritage/graph/2024-08-23/compressed/ 2024-08-23-compressed
# OR
swh datasets download-graph 2024-08-23