Dataverse Download Handler

A JavaScript implementation for seamlessly downloading files from a Dataverse repository directly from your website.

Overview

This system facilitates file downloads from a Dataverse data server by:

1. Allowing users to click on file links in HTML tables
2. Querying the Dataverse API to find the file ID
3. Generating a direct download URL
4. Initiating the browser download

File Structure

your-website/
├── index.html             # Your main HTML file with download tables
├── proxy_server.py        # CORS proxy server for local testing
├── setup_local_env.sh     # Script to set up local testing environment
├── assets/
│   ├── js/
│   │   ├── dataverse-download.js   # The download handler script
│   │   └── ... (other JS files)
│   └── ... (other asset folders)
└── test_dataverse.sh      # Shell script for testing/reference (not used in git repo and is just for reference)

Installation

1. Place dataverse-download.js in your website's assets/js/ directory
2. Add the script tag to your HTML file, before the closing </body> tag:

<script src="assets/js/dataverse-download.js"></script>

3. Ensure your HTML tables have links formatted like this: Use the href="#" to trigger the download and do not make this an actualy link to a file. The javascript will handle that.

<a href="#"><span style="color: rgb(0, 0, 0);">filename.zip</span></a>

How It Works

Download Logic

1. Link Setup: When the page loads, the script finds all table links with href="#" and registers click event listeners.

2. On Click: When a user clicks a download link:

   • The script extracts the filename from the link text
   • It queries the Dataverse API using the configured DOI
   • It searches for a file with a matching filename in the response
   • Once found, it gets the file ID
   • It constructs a direct download URL using this ID
   • It creates a temporary link element to initiate the download

3. Time-Sensitive URLs: The implementation handles Dataverse's time-sensitive URLs correctly:

   • The direct API URL (https://archive.data.jhu.edu/api/access/datafile/${fileId}) is permanent
   • When accessed, this URL generates a fresh time-sensitive S3 URL and redirects to it
   • This ensures that even if the page is left open for hours, when a user clicks a link, they get a freshly generated URL with its full validity period (15 minutes)

Shell Script vs. JavaScript Implementation

The file test_dataverse.sh is a proof-of-concept shell script that demonstrates the same logic but for command-line use:

The JavaScript implementation follows the same logic but in a web-browser context.

Configuration

In dataverse-download.js, configure your dataset's DOI:

// DOI for the dataset
const DOI = "10.7281/T1WF3NZ7";

Replace this value with your dataset's DOI.

Local Testing

To test the download functionality locally, you need to set up the local environment and run two servers:

0. Set Up Local Environment

First, run the setup script to download all required assets from the C3VDv2 repository:

chmod +x setup_local_env.sh
./setup_local_env.sh

This script will:

• Create the necessary folder structure (assets/bootstrap/css, assets/css, assets/img, assets/js, assets/data)
• Download CSS files (Bootstrap, custom styles)
• Download JavaScript files (jQuery, Bootstrap, load_csv_tables.js, video-modal.js)
• Download image assets
• Download CSV data files used by the tables

Note: You only need to run this script once, or whenever you want to update the assets from the remote repository.

1. Main Web Server

Start your main web server (serving the HTML files) on port 8000:

# Using Python 3
python3 -m http.server 8000

# Or using Python 2
python -m SimpleHTTPServer 8000

2. CORS Proxy Server

Due to CORS (Cross-Origin Resource Sharing) restrictions, the Dataverse API cannot be accessed directly from localhost. A proxy server is required to bypass these restrictions.

Start the proxy server on port 8001:

python3 proxy_server.py

The proxy server will:

• Accept requests on http://localhost:8001
• Forward them to the Dataverse API
• Add CORS headers to the response

Testing

1. Set up the environment (if you haven't already):

   ./setup_local_env.sh

2. Start both servers:

   • Main server on port 8000
   • Proxy server on port 8001

3. Open your browser and navigate to:

   http://localhost:8000
   

4. Click on any download link in the tables

5. The download should:

   • Show "Loading..." briefly
   • Fetch metadata from Dataverse via the proxy
   • Start the file download

Stopping the Servers

• Main server: Press Ctrl+C in the terminal where it's running
• Proxy server: Press Ctrl+C in the terminal where it's running, or kill the process:

  lsof -ti:8001 | xargs kill

Note on Production

In production (when deployed to a real web server), the proxy server may not be necessary if:

• The production server can make server-side requests to the Dataverse API
• Or if the Dataverse API allows CORS from your production domain

Troubleshooting

File Not Found

If you see "File not found in dataset" errors:

• Check the console log to see the list of available files, the filename is case sensitive.
• Ensure the filename in your HTML matches exactly with the filename in Dataverse
• Verify the DOI is correct

Download Not Starting

If the download doesn't start:

• Check browser console for errors
• Make sure pop-up blockers aren't preventing the download
• Verify network connectivity to the Dataverse server