Skip to main content

Dropbox & R Data

I'm always looking for ways to download data from the internet into R. Though I prefer to host and access plain-text data sets (CSV is my personal favourite) from GitHub (see my short paper on the topic) sometimes it's convenient to get data stored on Dropbox.

There has been a change in the way Dropbox URLs work and I just added some functionality to the repmis R package. So I though that I'ld write a quick post on how to directly download data from Dropbox into R.

The download method is different depending on whether or not your plain-text data is in a Dropbox Public folder or not.

Dropbox Public Folder

Dropbox is trying to do away with its public folders. New users need to actively create a Public folder. Regardless, sometimes you may want to download data from one. It used to be that files in Public folders were accessible through non-secure (http) URLs. It's easy to download these into R, just use the read.table command, where the URL is the file name. Dropbox recently changed Public links to be secure (https) URLs. These cannot be accessed with read.table.

Instead you need can use the source_data command from repmis:

FinURL <-"https://dl.dropbox.com/u/12581470/code/Replicability_code/Fin_Trans_Replication_Journal/Data/public.fin.msm.model.csv"

# Download data
FinRegulatorData <- repmis::source_data(FinURL,
                             sep = ",",
                             header = TRUE)

Non-Public Dropbox Folders

Getting data from a non-Public folder into R was a trickier. When you click on a Dropbox-based file's Share Link button you are taken to a secure URL, but not for the file itself. The Dropbox webpage you're taken to is filled with lots of other Dropbox information. I used to think that accessing a plain-text data file embedded in one of these webpages would require some tricky web scrapping. Luckily, today I ran across this blog post by Kay Cichini.

With some modifications I was able to easily create a function that could download data from non-Public Dropbox folders. The source_DropboxData command is in the most recent version of repmis (v0.2.4) is the result. All you need to know is the name of the file you want to download and its Dropbox key. You can find both of these things in the URL for the webpage that appears when you click on Share Link. Here is an example:

https://www.dropbox.com/s/exh4iobbm2p5p1v/fin_research_note.csv

The file name is at the very end (fin_research_note.csv) and the key is the string of letters and numbers in the middle (exh4iobbm2p5p1v). Now we have all of the information we need for source_DropboxData:

FinDataFull <- repmis::source_DropboxData("fin_research_note.csv",
                                  "exh4iobbm2p5p1v",
                                  sep = ",",
                                  header = TRUE)

Comments

K. Ram said…
You should also just try authenticating with Dropbox directly using my R package, rDrop.. Feedback welcome.
Unknown said…
Yeah, I only just now saw your rDrop package (I added a shout out to it in the repmis README). I'm definitely going to start using it.

Correct me if I'm wrong, but can you use rDrop db.read.csv to access data if you don't have the user's credentials?
Bob Muenchen said…
Thanks for the helpful post! If you find that rDrop does the same two examples (i.e. without authentication), it would make a great followup article.

Cheers,
Bob
Unknown said…
this seems to no longer be supported, unfortunately.
shivani said…
A bewildering web journal I visit this blog, it's unfathomably heavenly. Oddly, in this present blog's substance made purpose of actuality and reasonable. The substance of data is informative
Oracle Fusion Financials Online Training
Oracle Fusion HCM Online Training
Oracle Fusion SCM Online Training
PaperCoachNet said…
Great and useful article. Creating content regularly is very tough.Thanks you.Write more with reflective essay thesis.

basha said…
Excellent blog I visit this blog it's really awesome. The important thing is that in this blog content written clearly and understandable. The content of information is very informative.
Magnificent blog I visit this blog it's extremely wonderful. Interestingly, in this blog content composed plainly and reasonable. The substance of data is useful.
Oracle Fusion HCM Online Training
Oracle Fusion SCM Online Training
Oracle Fusion Financials Online Training
Big Data and Hadoop Training In Hyderabad
oracle fusion financials classroom training
Oracle Fusion HCM Classroom Training
oracle cpq online training / Oracle CPQ Class Room Training
Oracle Taleo Online Training
sailaja N said…
We are glad to announce that in COEPD we have introduced Digital Marketing Internship Programs (Self sponsored) for professionals who want to have hands on experience. In affiliation with IT companies we are providing this program. Presently, this program is available in COEPD Hyderabad premises. We deem in real time practical Internship program. We guide participants through real-time project examples and assignments, giving credits for Real-Time Internship. Our digital marketing certified mentors tutor our learning people through modules of Digital Marketing in an exhaustive manner. This internship is intelligently dedicated to our avid and passionate participants predominantly acknowledging and appreciating the fact that they are on the path of making a career in Digital Marketing discipline. We upskill and master the nitty-gritty of the Digital Marketing profession. More than a training institute, COEPD today stands differentiated as a mission to help you "Build your dream career" - COEPD way.
http://www.coepd.com/DMInternship.html
Data Science said…
Finally found very interesting blog with valuable information wafting for next blog update.
Data Analytics Course Online 360DigiTMG
Really nice and interesting article information shared was valuable, enjoyed reading this one. Thanks you.
Data Science Training in Hyderabad
You have completed certain reliable points there. I did some research on the subject and found that almost everyone will agree with your blog. PMP Certification in Hyderabad
Cyber Security said…
Nice Information Your first-class knowledge of this great job can become a suitable foundation for these people. I did some research on the subject and found that almost everyone will agree with your blog.
Cyber Security Course in Bangalore
Writing in style and getting good compliments on the article is hard enough, to be honest, but you did it so calmly and with such a great feeling and got the job done. This item is owned with style and I give it a nice compliment. Better!
Cyber Security Training in Bangalore
Really impressed! Everything is a very open and very clear clarification of the issues. It contains true facts. Your website is very valuable. Thanks for sharing.

Business Analytics Course in Bangalore
A good blog always contains new and exciting information and as I read it I felt that this blog really has all of these qualities that make a blog.

Data Analytics Course in Bangalore
TECHNOLOGY said…
I will really appreciate the writer's choice for choosing this excellent article appropriate to my matter. Here is deep description about the article matter which helped me more.
Data Science Course
Excelr Tuhin said…

keep up the good work. this is an Ossam post. This is to helpful, i have read here all post. i am impressed. thank you. this is our site please visit to know more information
data science courses
I am overwhelmed by your article with excellent topic and valuable information thanks for sharing.
Data Science Course in Bangalore
Data Science said…
Great article with unique information found useful looking forward for next blog thank you.
typeerror nonetype object is not subscriptable
Priyanka said…
Attend The Data Analyst Course From ExcelR. Practical Data Analyst Course Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Data Analyst Course.
Data Analyst Course
Top quality blog with unique content and information shared was valuable looking forward for next updated thank you.
Ethical Hacking Course in Bangalore
I have to search sites with relevant information ,This is a
wonderful blog,These type of blog keeps the users interest in
the website, i am impressed. thank you.
Data Science Training in Bangalore
Mallela said…
Very informative content and intresting blog post.Data science training in Mumbai
Actually I read it yesterday but I had some ideas about it and today I wanted to read it again because it is so well written.
Data Science Course in Vadodara
Mallela said…
Very informative content and intresting blog post.Data science course in Nashik
Hi, I looked at most of your posts. This article is probably where I got the most useful information for my research. Thanks for posting, we can find out more about this. Do you know of any other websites on this topic?
Data Science Course in Jaipur
Mallela said…
Very informative content and intresting blog.Data science course in Thiruvananthapuram
I was browsing the internet for information and found your blog. I am impressed with the information you have on this blog.
Data Science Course in Nagpur
Really impressed! Everything is a very open and very clear clarification of the issues. It contains true facts. Your website is very valuable. Thanks for sharing.
Data Science Course in Lucknow
Data Science said…
I am glad to discover this page. I have to thank you for the time I spent on this especially great reading !! I really liked each part and also bookmarked you for new information on your site.
Data Science Training in Chennai
Great post i must say and thanks for the information. Education is definitely a sticky subject. However, is still among the leading topics of our time. I appreciate your post and look forward to more.
Data Science Course in Bangalore
Very good message. I stumbled across your blog and wanted to say that I really enjoyed reading your articles. Anyway, I will subscribe to your feed and hope you post again soon.
Data Analytics course in Vadodara
Mallela said…
Thanks for posting the best information and the blog is very informative.Data science course in Faridabad
Fantastic Site with relevant information and content Shared was knowledgeable thank you.
Data Science Courses Hyderabad
Mallela said…
Thanks for posting the best information and the blog is very informative.Data science course in Faridabad
I am really enjoying reading your well written articles. It looks like you spend a lot of effort and time on your blog. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work.
artificial intelligence course in pune
Excellent Blog! I would like to thank for the efforts you have made in writing this post. I am hoping the same best work from you in the future as well. I wanted to thank you for this websites! Thanks for sharing. Great websites!
Data Science Training in Bangalore
Srigokul said…

Nice Blog. Thanks for Sharing this useful information...

Data science training in chennai
Data science course in chennai

Data Science said…
I am glad to discover this page. I have to thank you for the time I spent on this especially great reading !! I really liked each part and also bookmarked you for new information on your site.
Data Science Training in Chennai
I am a new user of this site, so here I saw several articles and posts published on this site, I am more interested in some of them, hope you will provide more information on these topics in your next articles.
data analytics training in bangalore
Tech said…
I really enjoy every part and have bookmarked you to see the new things you post. Well done for this excellent article. Please keep this work of the same quality.
Artificial Intelligence course in Chennai
Tech Science said…
Fantastic blog with excellent information and valuable content just added your blog to my bookmarking sites thank for sharing.
Data Science Course in Chennai
Excellent site with great content and very informative. I would like to thank you for the efforts you have made in writing.
Data Science Training in Bangalore
Fantastic Site with useful and unique content looking forward to the next update thank you.
Data Science Training in Hyderabad
I really enjoy reading all of your blogs. It is very helpful and very informative and I really learned a lot from it. Definitely a great article
Data Science Course Bangalore
Excellent site with great content and very informative. I would like to thank you for the efforts you have made in writing.
Data Science Training in Bangalore
Mallela said…
Thanks for posting the best information and the blog is very helpful.Data science course in Varanasi
AI Expert said…
I just got to this amazing site not long ago. I was actually captured with the piece of resources you have got here. Big thumbs up for making such wonderful blog page!
artificial intelligence course in pune
Pallavi reddy said…
i am glad to discover this page : i have to thank you for the time i spent on this especially great reading !! i really liked each part and also bookmarked you for new information on your site.
data science courses in hyderabad
madhavi reddy said…
I Want to leave a little comment to support and wish you the best of luck.we wish you the best of luck in all your blogging endeavors.
cyber security course in bangalore
I want to leave a little comment to support and wish you the best of luck.we wish you the best of luck in all your blogging enedevors.
best data science courses in hyderabad
I want to leave a little comment to support and wish you the best of luck.we wish you the best of luck in all your blogging enedevors.
data analytics course in bangalore
madhavi reddy said…
I Want to leave a little comment to support and wish you the best of luck.we wish you the best of luck in all your blogging endeavors.
data science in bangalore
Amazing information. I was looking for this one. Thanks a lot. I have a suggestion for the Best Digital Marketing Course in Janakpuri. If you want to enroll in Data Science Course, Join 99 Digital Academy as it offers Digital Marketing Course at an affordable Price. Click to Enroll Today.
Digital Marketing Course in Janakpuri.

Pallavi reddy said…
i am glad to discover this page : i have to thank you for the time i spent on this especially great reading !! i really liked each part and also bookmarked you for new information on your site.
best data science courses in bangalore
I want to leave a little comment to support and wish you the best of luck.we wish you the best of luck in all your blogging enedevors.
data analytics courses in bangalore
madhavi reddy said…
I Want to leave a little comment to support and wish you the best of luck.we wish you the best of luck in all your blogging endeavors.
data science certification in banagalore
Stupendous blog huge applause to the blogger and hoping you to come up with such an extraordinary content in future. Surely, this post will inspire many aspirants who are very keen in gaining the knowledge. Expecting many more contents with lot more curiosity further.

data science in bangalore
madhavi reddy said…
I Want to leave a little comment to support and wish you the best of luck.we wish you the best of luck in all your blogging endeavors.
data science course in bangalore with placement
Excellent Blog! I would like to thank for the efforts you have made in writing this post. I am hoping the same best work from you in the future as well. I wanted to thank you for this websites! Thanks for sharing. Great websites!
Data Science Training in Bangalore
I want to leave a little comment to support and wish you the best of luck.we wish you the best of luck in all your blogging enedevors.
data science in bangalore
I am a new user of this site, so here I saw several articles and posts published on this site, I am more interested in some of them, hope you will provide more information on these topics in your next articles.
data analytics training in bangalore
I need to thank you for this very good read and i have bookmarked to check out new things from your post. Thank you very much for sharing such a useful article and will definitely saved and revisit your site.
Data Science Course
Education said…
Your site is truly cool and this is an extraordinary moving article and If it's not too much trouble share more like that. Thank You..
Digital Marketing Institute in Bangalore
Unknown said…
Python Training in Bangalore Offered by myTectra is the most powerful Python Training ever offered with Top Quality Trainers, Best Price, Certification, and 24/7 Customer Care.

https://www.mytectra.com/python-training-in-bangalore.html
Institute said…
Wow, happy to see this awesome post. I hope this think help any newbie for their awesome work and by the way thanks for share this awesomeness, i thought this was a pretty interesting read when it comes to this topic. Thank you..
Artificial Intelligence Course
Knowledge said…
Thank a lot. You have done excellent job. I enjoyed your blog . Nice efforts
Cyber Security Course in Bangalore
InstituteBlr said…
I just got to this amazing site not long ago. I was actually captured with the piece of resources you have got here. Big thumbs up for making such wonderful blog page!
data analytics course in bangalore
I want to leave a little comment to support and wish you the best of luck.we wish you the best of luck in all your blogging enedevors.
data science course fees in bangalore
I want to leave a little comment to support and wish you the best of luck.we wish you the best of luck in all your blogging enedevors.
data science in bangalore
Great post i must say and thanks for the information. Education is definitely a sticky subject. However, is still among the leading topics of our time. I appreciate your post and look forward to more.
Data Science Course in Bangalore
trainingcourses said…
Pretty good post. I just stumbled upon your blog and wanted to say that I have really enjoyed reading your blog posts. Any way I’ll be subscribing to your feed and I hope you post again soon.
business analytics course
Pallavi reddy said…
i am glad to discover this page : i have to thank you for the time i spent on this especially great reading !! i really liked each part and also bookmarked you for new information on your site.
cyber security training in bangalore
Unknown said…
I want to leave a little comment to support and wish you the best of luck.we wish you the best of luck in all your blogging endeavours.
data science course in bangalore with placement
Great post i must say and thanks for the information. Education is definitely a sticky subject. However, is still among the leading topics of our time. I appreciate your post and look forward to more.
Data Science Course in Bangalore
Pallavi reddy said…
i am glad to discover this page : i have to thank you for the time i spent on this especially great reading !! i really liked each part and also bookmarked you for new information on your site.
cyber security training in bangalore
Data Science said…
I am glad to discover this page. I have to thank you for the time I spent on this especially great reading !! I really liked each part and also bookmarked you for new information on your site.
Data Science Training in Chennai
I want to leave a little comment to support and wish you the best of luck.we wish you the best of luck in all your blogging enedevors.
data analytics courses in bangalore
Mallela said…
Thanks for posting the best information and the blog is very helpful.artificial intelligence course in hyderabad
Pallavi reddy said…
i am glad to discover this page : i have to thank you for the time i spent on this especially great reading !! i really liked each part and also bookmarked you for new information on your site.
cyber security training in bangalore
I feel very grateful that I read this. It is very helpful and very informative and I really learned a lot from it.
Data Science Course in Chennai
Unknown said…
I want to leave a little comment to support and wish you the best of luck.we wish you the best of luck in all your blogging enedevors.
data analytics courses in bangalore
trainingcourses said…
I wanted to leave a little comment to support you and wish you a good continuation. Wishing you the best of luck for all your blogging efforts.

best data science institute in hyderabad
Technology said…
All of these posts were incredible perfect. It would be great if you’ll post more updates and your website is really cool and this is a great inspiring article.
Artificial Intelligence course in Chennai
InstituteBlr said…
I just got to this amazing site not long ago. I was actually captured with the piece of resources you have got here. Big thumbs up for making such wonderful blog page!
data analytics course in bangalore
trainingcourses said…
This is an excellent post . thanks for sharing it. It is really what I wanted to see. I hope in the future you will continue to share such an excellent post.

digital marketing courses in hyderabad with placement
Pallavi reddy said…
i am glad to discover this page : i have to thank you for the time i spent on this especially great reading !! i really liked each part and also bookmarked you for new information on your site.
data scientist course in bangalore
princika said…
I want to leave a little comment to support and wish you the best of luck.we wish you the best of luck in all your blogging enedevors.
data science course fees in bangalore
Data Science said…
I am glad to discover this page. I have to thank you for the time I spent on this especially great reading !! I really liked each part and also bookmarked you for new information on your site.
Data Science Training in Chennai
I am a new user of this site, so here I saw several articles and posts published on this site, I am more interested in some of them, hope you will provide more information on these topics in your next articles.
data analytics training in bangalore
Mallela said…
Thanks for posting the best information and the blog is very important.data science institutes in hyderabad
Education said…
Your site is truly cool and this is an extraordinary moving article and If it's not too much trouble share more like that. Thank You..
Digital Marketing Course in Hyderabad
You have completed certain reliable points there. I did some research on the subject and found that almost everyone will agree with your blog.

Data Science Training in Bangalore
I have voiced some of the posts on your website now, and I really like your blogging style. I added it to my list of favorite blogging sites and will be back soon ...

Digital Marketing Training in Bangalore
I found Habit to be a transparent site, a social hub that is a conglomerate of buyers and sellers willing to offer digital advice online at a decent cost.

Artificial Intelligence Training in Bangalore
The Extraordinary blog went amazed with the content that they have developed in a very descriptive manner. This type of content surely ensures the participants explore themselves. Hope you deliver the same near the future as well. Gratitude to the blogger for the efforts.

Machine Learning Course in Bangalore
Excellent Blog! I would like to thank you for the efforts you have made in writing this post. Gained lots of knowledge.
Data Analytics Course
AI Courses said…
What an incredible message this is. Truly one of the best posts I have ever seen in my life. Wow, keep it up.
AI Courses in Bangalore
Awesome article. I enjoyed reading your articles. this can be really a good scan for me. wanting forward to reading new articles. maintain the nice work!
Data Science Courses in Bangalore
I am sure it will help many people. Keep up the good work. It's very compelling and I enjoyed browsing the entire blog.
Business Analytics Course in Bangalore
trainingcourses said…
Excellent effort to make this blog more wonderful and attractive.
best data science institute in hyderabad
Pallavi reddy said…
i am glad to discover this page : i have to thank you for the time i spent on this especially great reading !! i really liked each part and also bookmarked you for new information on your site.
artificial intelligence training in chennai
Data Science said…
I am glad to discover this page. I have to thank you for the time I spent on this especially great reading !! I really liked each part and also bookmarked you for new information on your site.
Data Science Training in Chennai

Popular posts from this blog

A Link Between topicmodels LDA and LDAvis

Carson Sievert and Kenny Shirley have put together the really nice LDAvis R package. It provides a Shiny-based interactive interface for exploring the output from Latent Dirichlet Allocation topic models. If you've never used it, I highly recommend checking out their XKCD example (this paper also has some nice background). LDAvis doesn't fit topic models, it just visualises the output. As such it is agnostic about what package you use to fit your LDA topic model. They have a useful example of how to use output from the lda package. I wanted to use LDAvis with output from the topicmodels package. It works really nicely with texts preprocessed using the tm package. The trick is extracting the information LDAvis requires from the model and placing it into a specifically structured JSON formatted object. To make the conversion from topicmodels output to LDAvis JSON input easier, I created a linking function called topicmodels_json_ldavis . The full function is below. To

Set up R/Stan on Amazon EC2

A few months ago I posted the script that I use to set up my R/JAGS working environment on an Amazon EC2 instance. Since then I've largely transitioned to using R/ Stan to estimate my models. So, I've updated my setup script (see below). There are a few other changes: I don't install/use RStudio on Amazon EC2. Instead, I just use R from the terminal. Don't get me wrong, I love RStudio. But since what I'm doing on EC2 is just running simulations (I handle the results on my local machine), RStudio is overkill. I don't install git anymore. Instead I use source_url (from devtools) and source_data (from repmis) to source scripts from GitHub. Again all of the manipulation I'm doing to these scripts is on my local machine.

Slide: one function for lag/lead variables in data frames, including time-series cross-sectional data

I often want to quickly create a lag or lead variable in an R data frame. Sometimes I also want to create the lag or lead variable for different groups in a data frame, for example, if I want to lag GDP for each country in a data frame. I've found the various R methods for doing this hard to remember and usually need to look at old blog posts . Any time we find ourselves using the same series of codes over and over, it's probably time to put them into a function. So, I added a new command– slide –to the DataCombine R package (v0.1.5). Building on the shift function TszKin Julian posted on his blog , slide allows you to slide a variable up by any time unit to create a lead or down to create a lag. It returns the lag/lead variable to a new column in your data frame. It works with both data that has one observed unit and with time-series cross-sectional data. Note: your data needs to be in ascending time order with equally spaced time increments. For example 1995, 1996