Moja global: Interactive Visualisation Tool for GCBM Output

Warm greetings to the OSC Hub community.

My name is Abhineet, and it brings me great pleasure to share with you that I’ll be working with @project-mojaglobal for this year’s Google Summer of Code.

Thank you so much to all the @gsoc-admins and @Guy for accepting my proposal!

Here is my first weekly update…

Community Bonding Begins!

To start things off, @Guy organized a video conference a few days after the announcement. It was a first of its kind experience for me. To be conversing with so many nice people from so many different parts of the world!

But the highlight of that meeting was surely when all of sudden a cute little member of our meeting announced, “we have guinea pigs!” :smile:

After the meeting, we received our first assignment from @Guy:

To encourage communication between the students, I would like you to discuss between the four of you and inform us about how you want to communicate your plans and progress.

Lead by @abhi211199, we spent rest our time discussing (quite constructively) this. We were able to arrive at a consensus within the provided deadline and submitted our first deliverable!

The most important thing that happened since the announcement is that I made my first contact with my wonderful mentor @koukas.

The key takeaway for me from all the interactions that I’ve had till now is that – pun unintended – I’m in good company!

Looking forward to the journey ahead!

5 Likes

This is the result of the stand-up for this week.

Last week was successful. The students decided where and how to post their work. The approach is to make comms as efficient as possible. We will post weekly blogs here. We will duplicate them on a specially created repo on our GitHub. We will also use GitHub to post our work plan and progress for each item. We will have weekly deliverables (issues) that will be resolved. So the repo will have all info.

This week, the focus will be on getting a work plan in place by the end of the week. This will require close communication between the student and the mentor. We are still figuring out how it will work precisely.

The challenge this week will clearly be to get the communication to work. There is some reluctance on the side of the students to get the attention of the mentors. They should be not hesitate to get attention for their queries. If necessary they can use other channels of communication or ask other mentors or admins to assist.

But overall we are off on a great start. Looking forward to the work plans!

2 Likes

Week 20: Project Plan, Hardware Failure & A Choropleth

This week had promised me a project plan but instead it started with the demise of my laptop’s hard drive. But thankfully, it didn’t leave me empty handed. In, fact it left me with a sense of optimism for the future!

Summer Strikes My Hard Drive

My HDD had been telling me of failing sectors but I had been overlooking it. I understood the gravity of those warnings but still ignored them. Fortunately, this oversight didn’t cause me any significant damage. All thanks to my very resourceful father, who was able to procure a SDD (yes, a SSD!) in a lock-down!

All this did cost me a day or two but nevertheless I was able to publish a preliminary project plan. I’ll further refine it in the coming week and will clearly label my deliverables as I further understand the project.

Now, I’m vigilantly monitoring the temperature of my hard drive so that another failure doesn’t happen in the near future.

Experimenting with D3.js

To get a feel of how I can combine spatial and non-spatial data, I made this choropleth map of my home state of Chhattisgarh:

It depicts how much percentage of the state’s total ground water is available in a district. I made it in an Observable notebook using the D3.js library. Converted the files available here for the map and got the ground water availability data from here.

Communicate! Communicate Communicate!

As I was feeling uncertain about the future of my project, I contacted my mentors. Their feedback was very helpful. It helped me calm down and gave me a much needed sense of direction. I’m extremely grateful to @mfellows and @koukas. Thank you so much guys!

All in all, it was a bittersweet week but I did learn how immensely valuable open communication is.

3 Likes

Abhineet,
I would not have wanted to be in your shoes this week. I am so happy it all ended well. Still impressive that you got this great map done! Good stuff.
I also like your project page on GitHub. I think you are the only one who got the contributors-script working. :+1:t4:
I am looking forward to your final workplan.

2 Likes

Week 21: Raster Graphics & Python Hell

This week was spent trying to understand and evaluate the technical aspects of the project. Which is just a fancy way of saying that I spent the week installing and testing a bunch of Python frameworks :smile:

I’ve landed on these two frameworks that I think would be great for this project:

  • Dash — For creating the web interface purely in Python.
  • Terracotta – A tile server for optimizing and serving the raster graphics.

I’ve shared this with my mentors and they’re looking into it and hopefully I’ll start coding by the end of May!

What’s a GeoTIFF?

While learning about this project, I dived into the fascinating ocean of computer cartography. Although I didn’t see the bottom of this very deep ocean (and probably never will) but I sure did learn a lot!

It’s a very well organized community with the primary actor being the Open Source Geospatial (OSGeo) Foundation and the most important project being the Geospatial Data Abstraction Library (GDAL).

moja global’s primary project, FLINT, heavily uses GDAL for all geo-spatial calculations. Apart from that GDAL is also used for producing the GeoTIFF raster graphics that’ll be the primary input of my project.

Created at NASA, the GeoTIFF format allows researchers all over the world to store and share geo-referenced map data. They’re not very memory efficient on their own but you can significantly reduce their size by using compression algorithms like DEFLATE.

Even with compression, however, it can be hard to work direclty with GeoTIFFs in the browser. Yes, there is [geotiff.js] but, quite paradoxically, I couldn’t get it to work with compressed files. Anyhow, in projects like mine, it’s best to use a tile servers like MapServer or Terracotta.

Frameworks are Awesome (but their installation is not)

Frameworks make the job of the software developer very easy, if not outright trivial! But they come with the cost of being a hard dependency. Now, that might not look like a very high cost at first glance but when you try to get it installed, the cost becomes very clear, especially in Python. I’ll let this xkcd speak for me:

Community Bonding Continues

Apart from all this, near the end of this week we had a group video call. It was a short exchange but nevertheless a vital one where all the team members were asked for their opinions on what’s going well and what needs work. We identified that our communication medium (Slack) is great and that moja global’s documentation needs work.

3 Likes

Abhineet,
I really enjoyed reading your blog. It is really good and useful beyond people that know moja global. Well done.
Also good job on the frameworks. I am confident that we will be ready to start coding by the end of the week.
Exciting!

2 Likes

Week 22: Writing a Pitch & Contributing Contribution Guidelines

This week, I wrote a pitch and contributed to moja global’s GitHub contribution guidelines.

A Pitch

It is important that my mentors understand what I plan on doing. But the more important thing is that I should understand my decisions. This is why, to explain my rationale behind using Terracotta, I wrote a pitch.

While writing it, I realized that there were many gaps in my knowledge. For example, I hadn’t realized that Terracotta is not exactly a framework. It is a CLI tool. This would’ve probably set me off, if I had discovered this in the future.

I’m grateful to my mentor’s questions. They made me realize that I should clarify my plan. As a result, now I have a better understanding of Terracotta. How I will integrate it into my project is more clear now.

CONTRIBUTING

On, GitHub, the contribution guidelines of a project are stored in a file named CONTRIBUTING.md. GitHub points out to this file at various stages when you try to make a contribution. So, this file (along with the README) plays a very important role in promoting contributions.

moja global’s contribution guidelines were divided into multiple documents. They were then linked in CONTRIBUTING. I realized that I can merge these files. So, I created an issue and then submitted a PR. It got merged :tada:

I hope that this will help in increasing contributions to moja global.

1 Like

Abhineet,
Thank you for the Pitch. I think we all learned from your rationale in relation to Terracotta.
Also great that you contributed to the contributing guidelines. Good job. You have earned your badge on the landing page! Now we hope we will get many contributions as a results. Keep up the great work.

1 Like

Week 23: Application Architecture & Coding Begins!

This week I shared the software architecture of my project and finally started coding!

Modern Python Development

Before setting up my project for coding, I did some research and found out about all the modern Python development tools that are available. Here’s the tech stack that I ended up choosing:

IDE: Visual Studio Code
Code Linting: Pylint
Code Formatting: autopep8
Dependency Management: Poerty
Python Version Management: Pyenv

Apart from that, I’m using Pytest as my testing framework.

I’ve selected these tools to ensure that development is fun and easy. Checkout my project’s README. It briefly describes how anyone can get up and running and start hacking on Taswira (the tool that I’m creating).

Taswira’s Architecture Diagram

The application architecture conveys the underlying technical aspects of the application to various stakeholder in an easy-to-understand manner. There are a lot of standards that you can use to visualize your software’s architecture. The one I used it called the C4 Model. I used the Structurizr software to create the diagram.

1 Like

Abhineet,
I love the name Taswira, you should mention that it is the word for “visualize” in Swahili. I suppose you are following the selection of the word “moja” as it is Swahili for “one”. Cool choice. I really like it.
Great job to with the architecture for the visualisation tool. You have a good balance between documenting your work neatly and making progress on the actual code to be developed. Well done!

1 Like

Week 24: Ingestion support

This week, I started work on an important component of my back-end. I’ll be honest, this week I didn’t feel very productive. But I’m not gonna let this demotivate me.

From here on I’ll focus on building good habits and following a strict routine. I believe that this can help me be sustainably motivated to deliver this project.

About “Taswira”

Following @Guy’s suggestion from above, I would like to write a few lines about my project’s name.

In the OSS world, it a very common practice to name your project. So, choosing a nice name for my project was one the first thing that was on my mind when I found out that I’ve been selected :smile:

In my initial research for GSoC, I had learned from the website’s about page that “moja” was Swahili for “one”. So, following moja, I went to Google Translate.

I found out that Swahili for “visualization” – Taswira – is also a word in my native language (Hindi)! To add to this delight, the meaning is also almost same in both the languages!

See you next Friday. Heri!

1 Like

Hi Abhineet,

I did not know that Taswira has the same meaning in Hindi. This sounds like the perfect name for your project.

I hope this week will be more productive. I have the impression that you are well on your way to get good results. So I am looking forward to your blog next Friday to get an update.

Keep going! Thank you for your commitment!

1 Like

Week 25: Metadata Support

This week, I worked on my Terracotta-based back-end. I added support for associating metadata to a raster.

I still need to add tests for the metadata part but other that that my back-end is almost finished.

The best thing about this week is that now I’ve started following a more consistent schedule. I’m waking up early and exercising more consistently. I believe that this will be beneficial for me in general and not only in GSoC.

This week we also had one of our regular GSoC group video calls. I’m really grateful that moja global’s members actively look for and are incredibly receptive to our feedback.

Also, these past few weeks there has been a flurry of activity in our internal Slack. A lot of new people have joined. I’m extremely delighted to see this influx of contributors. I’m sure that their contributions will be beneficial for moja!

1 Like

Abhineet,
Good progress as the first period of coding is coming to an end. I like the personal touch in your blog. I also like the updated plan as in an agile manner you are adjusting your available time to the changing circumstances while keep the end-goal in mind. Let us know how the tests with Terracotta go.

1 Like

Thank you, Guy. The tests went great. I’ve updated my project plan to reflect the progress, accordingly.

This week’s blog post…

Week 26

This week, I completed the first part that I had planned for the coding phase. On the way, I learned a lot about testing. Let me share some of my insights with you.

Tests & Testing

Every programmer, tests their program. The way in which a person tests can differ. Without using software engineering jargon, tests may be informally divided into manual test and automated tests.

In the former, the developer manually tries out their implementation whereas, in the latter they write test cases. These cases automatically invokes the appropriate procedure and then tests it.

From my experience these past weeks, I can tell you that testing is hard. Sometimes, it’s actually more difficult to write the test case than to write the program that you’re testing.

Having said that, I still believe that in the end the tests are worth it. It’s because they pave the way for me to do interesting activities in the future like continuous integration. But most importantly, tests that are well-written, will allow me to refactor my code with ease.

1 Like

Week 27

After weeks of working on CLI related parts, this week I started working on some GUI.

It’s the front-end of my tool. I’m writing it entirely in Python using a framework called Dash. As of now, I’ve added three major components to it:

  • A map that has a raster file overlayed over it.
  • A drop-down.
  • A slider with years marked beneath it.

All these components work in conjunction with each other. Interacting with one, automatically updates the other. The cool thing about that is the way in which its implemented. Instead of manually updating the values of these components, I utilized the power of Dash Callbacks. These constructs are the main selling-point of Dash. They enable what’s called as Reactive Programming in Dash. Combine this with a variety of fancy developer features like hot-reloading, in-browser error messages, etc., and developing a UI becomes enjoyable!

I wasn’t able to share screenshots with you because, the layout isn’t properly polished yet. I’ll get it sorted by next week so that I can share some of my work with you :slight_smile:

Have a nice weekend :v:

1 Like

Hi Abhineet,
You passed the first period! Well done!!!
And you are starting a new phase of development. This is exciting. I am looking forward to the nice screenshots you are going to share.
Good job, keep going!

1 Like

Thanks, Guy!

Week 28

Continuing to work on the UI, this week I added another component: A graph that shows the change in value of the selected ecosystem indicator over the years.

Here’s the screenshot that I promised last week:

I’ve annotated it to describe the different components:

  1. A preview of the raster file, for the currently selected indicator (“NBP” in the screenshot), overlaid onto a map.
  2. A drop-down containing the list of all ecosystem indicators for which data is available.
  3. A slider of all the years for which indicator data is available.
  4. A graph showing how the value of the currently selected indicator changed over the years.

I glad that my mentors are happy with my work so far. They suggested that I should add a legend on the map that explains what the different shades in the raster preview represent. They also suggested that I should add a play button so that users can automatically step through the different years.

I think both of these are great suggestions and I’ll try to incorporate them over the course of the next week.

See you next week :slight_smile:

2 Likes

Hi Abhineet,
Nice work. I really like the visual. We will use it for the article on the moja global website. Your visualisation is coming together nicely.

1 Like

Week 29

This week I made a major improvement by adding support for a palette. This means that the raster overlays are now more colorful :rainbow:

Accompanying this change was the addition of a new component: a map legend that indicates the value to which the different shades of the palette correspond to.

Apart from that, I also made some bug fixes. Most importantly, though, I added OpenStreeMap attribution notice on the map; something that I had forgotten before :sweat_smile:

I had also planned on adding a play button, that automatically cycles through different years, but couldn’t do it this week. I’ll start thinking and prototyping it right away and will surely show you something by my next post.

May you have a nice weekend.

1 Like