filltex: automagically fill LaTex bibliography

Check me out on github.com/dgerosa/filltex.

filltex is a simple tool to fill LaTex reference lists with records from the ADS and INSPIRE databases. ADS and INSPIRE are the most common databases used among the astronomy and theoretical physics scientific communities, respectively. filltex automatically looks for all citation labels present in a tex document and, by means of web-scraping, downloads all the required citation records from either of the two databases. filltex significantly speeds up the LaTex scientific writing workflow, as all required actions (compile the tex file, fill the bibliography, compile the bibliography, compile the tex file again) are automated in a single command. We also provide an integration of filltex for the macos LaTex editor TexShop.

If you use filltex for your research, please drop a citation to this paper:

filltex: Automatic queries to ADS and INSPIRE databases to fill LaTex bibliography, Davide Gerosa, Michele Vallisneri, The Journal of Open Source Software 2 (2017) 13.

Of course, you can use filltex to cite filltex! Just put \cite{2017JOSS....2..222G} in your tex file!

Installation

filltex can be installed from the python package index Pypi:

pip install filltex

If you're a TexShop user and want to use this feature, run

filltex install-texshop

If you want to give it a try, you can run it on the example.tex file provided in this repository:

git clone https://github.com/dgerosa/filltex.git
cd filltex/example
filltex example

and you should get a filled .bib file and a finished .pdf.

What's about?

What happens when you compile a LaTex file? How's bibliography handled?

Run pdflatex and all requested citation keys are dumped into a .aux file.
You should have the required entries in you .bib file.
Run bibtex, which looks for citations inside the .bib file and writes the results into a .bbl.
Run pdflatex again, which processes the .bbl into the compiled paper, and dumps the citation records into .aux.
Finally run pdflatex again, which puts the correct citation records into the paper.

The commands you need to run are: pdflatex, bibtex, pdflatex, pdflatex. These, of course can be put into a script or a makefile and done in one goal. filltex is meant to automatically solve the second point as well: look for citations on ADS, INSPIRE or both.

So, here is the deal:

The fillbib python script queries both databases and creates/updates a .bib file without getting each record manually.
The filltex bash script puts everything together to go from a .tex (and no .bib) into a .pdf.
I also provide TexShop engines for mac users

Of course, all of this works if your citations are specified in the ADS or INSPIRE format, e.g. \cite{2016PhRvL.116f1102A}, \cite{Abbott:2016blz}. If you use your personal keys \cite{amazing_paper}there's no way to get them from a database.

Usage

fillbib (script)

fillbib has two working modes. It can either look for citations into a .aux file and create/update a bibtex file with the records found on ADS and INSPIRE, or it can fetch a list of bibtex entries specified from the command line from ADS or INSPIRE.

usage: fillbib.py [-h] [--generate] [--journal_arXiv_fallback]
                  [--max-num-authors MAX_NUM_AUTHORS]
                  [--num-authors-short NUM_AUTHORS_SHORT]
                  {tex,list} ...

positional arguments:
  {tex,list}            Subcommands
    tex                 Create a bibliography for a tex document
    list                Create a bibliography given a list of ADS/iNSPIRE keys

optional arguments:
  -h, --help            show this help message and exit
  --generate            Generate the BibTeX entries from the metadata (this is
                        useful to customize the generated BiBTeX file)
  --journal_arXiv_fallback
                        Set the journal entry to be arXiv for unpublished
                        preprints (iNSPIRE entries only, requres --generate)
  --max-num-authors MAX_NUM_AUTHORS
                        Include at most this many authors for each bibtex
                        entry(iNSPIRE entries only, requres --generate)
  --num-authors-short NUM_AUTHORS_SHORT
                        Number of authors to list if the number of authors is
                        larger than max_num_authors(iNSPIRE entries only,
                        defaults to max-num-authors, requres --generate)

The first argument specifies the subcommand to run.

tex will produce a bibtex file given an .aux file
list will print a bibtex file given a list of keys from CLI

The help for the two subcommands can be obtained with

fillbib.py {tex,list} --help

When working in tex mode it is possible to specify the name of the bibtex file using the option --bibtex. Otherwise, the code will scan the .aux file to guess the name of your bibliography file. Arguments can be typed with or without extension, and the script is smart enough to figure it out. You need to have .aux file already, not just the .tex. If you don't have it, run pdflatex once.

fillbib.py contains two short unit tests, to make sure the web-scarping part is done correctly. You can run them from the filltex directiory using

python
> import fillbib
> fillbib.test_ads()
> fillbib.test_inspire()

or simply using pytest

pytest fillbib

fillbib supports both python 2 (2.6 or higher) and python 3.

filltex (script)

filltex does the whole thing: compiles LaTex, fills the bibliography and gives you the final .pdf. Usage:

filltex <tex file>

Argument can be with or without extension, and the script is smart enough to figure it out.

Since ADS bibliography items contains journal abbreviations, you need to use aas_macros.sty (available here). Don't worry, you just put \include{aas_macros} in your .tex file, and filltex will download the file for you if you need it.

At the end, filltex also runs TexCount which counts the words in your document.

TexShop

I use the TexShop editor, so I wrote an implementation of filltex for it. If you copied the filltex.engine file as specified above, just open your paper with TexShop and select filltex from the drop menu on the left. Now automagically compile your paper with Typeset or cmd-T.

Example

A short example.tex file is provided, where you can try this new way of writing papers!

Known limitations

Treating arXiv e-prints with ADS is tricky. When an e-print gets published they change the database key, but make the old key point to the new version! For instance, the key switches from 2016arXiv160203837T to 2016PhRvL.116f1102A. If you're citing an e-print which is not yet published, everything is fine: only the arXiv key (e.g. 2016arXiv160203837T) is available and your reference list will show the arXiv version. If you're citing a paper that is published, both the e-print key (e.g. 2016arXiv160203837T) and the published-version key (e.g. 2016PhRvL.116f1102A) are available. When used, they will both point to the same published version! If you write a document with citations to both, this will cause the same record to appear twice in your reference list (see the example file). To avoid the issue, always use the published-paper key if a published version is out. INSPIRE doesn't have this problem, because they don't change the citation key when a paper gets published.
Multiple bibliographies are not allowed, only one .bib file per paper. I don't plan to implement multiple bibliographies in here, because you're not going to need them with this script: one paper, one bibliography, that's all.

Manual installation from repository

If you don't like pip (but why wouldn't you?), you can install the code manually:

git clone https://github.com/dgerosa/filltex.git # Clone repo
cd filltex
chmod +x bin/* # Make bin content executable
PATH=$PATH:$(pwd)/bin # Add bin directory to path
echo "PATH=$PATH:$(pwd)/bin" >> ${HOME}/.bashrc # To add the new path to your .bashrc    
cp filltex.engine ~/Library/TeXshop/Engines/filltex.engine # To install the Texshop engine

filltex uses TexCount, which is included in most Tex distribution. In case it's not in yours, here you can find installation instruction.

References to filltex

filltex is included in the suggested tools from the INSPIRE team.
filltex is included the official v3.80 release of Texshop.

Credits

The code is developed and maintained by Davide Gerosa. If you find bugs, want to contribute to this project (any help is welcome!) or need help with it, just open an issue here on GitHub.

The idea started from this python course taught by Michele Vallisneri at Caltech (and in particular from this example) and was later developed with key contributions from David Radice. We also thank Lars Holm Nielsen, reviewer for The Journal of Open Software, for several suggestions which improved filltex. TexCount is developed by Einar Andreas Rodland. Useful info on the INSPIRE and ADS APIs are available here and here.

Changes

v1.0: Initial release, main functionalities.

v1.1: Version accepted in JOSS.

v1.2: Uploaded on pip.

v1.3: Compatible with new ADS "Bumblebee".

v1.4: Compatible with new INSPIRE API.

v1.5: New tex and list subcommands.