biologist with a computing problem; personal account @__crusoe
22 stories
·
1 follower

Uploading Metadata: A Case Study in Adjusting Undocumented Behavior on Legacy Code Bases

1 Share
Reading Time: 4 minutes

I sometimes live stream my development work on the NASA Landsat Data Processing Pipeline. It’s an asychronous task management app written in Python.

NASA Vehicle Assembly Building with Launch Pad 39A in the Background.
NASA Vehicle Assembly Building with Launch Pad 39A in the Background. Photo by @mgde_visuals.

The pipeline’s tasks include:

  • collecting and unpacking massive data collected by Earth-orbiting satellites about the radiation reflected from Earth’s surface at various wavelengths
  • turning that data into image representations that the human eye can interpret
  • manipulating those images, say, to emphasize a particular wavelength, or divide the image into tiles for easier analysis
  • uploading the resulting images to the Zooniverse, a platform for people-powered scientific research.

Sound cool? It is. If you’d like to learn more about it and/or enjoy the live streams where I write code for it, I recommend taking a look at this technical briefing I made for you. I explain what the app does and why it matters, then I guide you through the code…all while sipping coffee, bikini-clad, in front of a gorgeous Space Coast sunrise 😉.

The technical briefing done, it was time to get to work.

What you see below are the recordings of about two hours of streams. In these streams, I’m working on including additional data with the images uploaded to the Zooniverse—data that records the latitude and longitude coordinates where the image was taken, the size of the image, and other information.

You get to see me make design decisions about how our app should work for Earth science research teams that need metadata uploaded versus teams that don’t.

You also get to see me spelunk through a fairly complex app to identify:

  • What’s already working
  • Where functionality is missing
  • Where new functionality should live

In the past I’ve done detailed show notes for these streams, but nowadays I’m streaming enough that I don’t have time to do that. Instead, here’s how this will go:

  1. Eventually if enough people support the Patreon I’ll hire someone to help create show notes so you get them again 😉
  2. In the meantime, I will pull out enticing quotes from the streams to reeeel you into watching them. That’s right—just like you’re a fish.

6:23 – This is where I explain the decisions we (well, I, with you as an audience) have to make about how to make this app work for multiple potential use cases.

59:20 – “GAH! LOOK AT THIS!”

1:04:00 – “So we may need to parse a CSV which…I’m not super pumped about…but whatever”

1:17:30 – “So we open, blah blah blah, blah blah blah, then we do this dict reader AND THEN if we were to print row it would look like…this, supposedly.”

1:26:50 – “I want to make reads fast, because we only have to do [the insertion] once, but we have to read constantly.”

Here’s some additional footage after we took a break:

2:30 – “And then I don’t ever set any config. HelLO?…No…NO…..THERE we go!”

8:08 – Go here if you want to hear me sing a Backstreet Boys song for some reason

10:55 – Oh, this  was probably the reason.

We left off on this stream asking the research team for the wgs row of a location with a visible coast, which I ended up solving by using the Landsat Acquisition Map to locate a tile that likely included coastline and get the WRS (World Reference System) row and path for that tile.

While I finished this feature up off-stream, in future streams we’ll look at details of UI design, deployment, and maybe even some researcher-facing features!

If you liked this piece, you might also like:

The debugging posts (a toolkit to help you respond to problems in software)

The Listening Series (Prepare to question much of what you know about how to be good at your job.)

Skills for working on distributed teams (including communication skills that will make your job easier)

 

Read the whole story
biocrusoe
29 days ago
reply
Lansing, MI
Share this story
Delete

SIMDe 0.5.0 Released

2 Shares

I’m pleased to announce the availability of the first release of SIMD Everywhere (SIMDe), version 0.5.0, representing more than three years of work by over a dozen developers.

SIMDe is a permissively-licensed (MIT) header-only library which provides fast, portable implementations of SIMD intrinsics for platforms which aren’t natively supported by the API in question.

For example, with SIMDe you can use SSE on ARM, POWER, WebAssembly, or almost any platform with a C compiler. That includes, of course, x86 CPUs which don’t support the ISA extension is question (e.g., calling AVX-512F functions on a CPU which doesn’t natively support them).

If the target natively supports the SIMD extension in question there is no performance penalty for using SIMDe. Otherwise, accelerated implementations, such as NEON on ARM, AltiVec on POWER, WASM SIMD on WebAssembly, etc., are used when available to provide good performance.

SIMDe has already been used to port several packages to additional architectures through either upstream support or distribution packages, particularly on Debian.

If you’d like to play with SIMDe online, you can do so on Compiler Explorer.

What is in 0.5.0

The 0.5.0 release is SIMDe’s first release. It includes complete implementations of:

  • MMX
  • SSE
  • SSE2
  • SSE3
  • SSSE3
  • SSE4.1
  • AVX
  • FMA
  • GFNI

We also have rapidly progressing implementations of many other extensions including NEON, AVX2, SVML, and several AVX-512 extensions (AVX-512F, AVX-512BW, AVX-512VL, etc.).

Additionally, we have an extensive test suite to verify our implementations.

What is coming next

Work on SIMDe is proceeding rapidly, but there are a lot of functions to implement… x86 alone has about 6,000 SIMD functions, and we’ve implemented about 2,000 of them. We will keep adding more functions and improving the implementations we already have.

Our NEON implementation is being worked on very actively right now by Sean Maher and Christopher Moore, and is expected to continue progressing rapidly.

We currently have two Google Summer of Code students working on the project as well; Hidayat Khan is working on finishing up AVX2, and Himanshi Mathur is focused on SVML.

If you’re interested in using SIMDe but need some specific functions to be implemented first, please file an issue and we may be able to prioritize those functions.

Getting Involved

If you’re interested in helping out please get in touch. We have a chat room on Gitter which is fairly active if you have questions, or of course you can just dive right in on the issue tracker.

Read the whole story
luizirber
42 days ago
reply
Davis, CA
biocrusoe
42 days ago
reply
Lansing, MI
Share this story
Delete

Top tech grads are increasingly unwilling to work for Big Tech, viewing it as a new, unethical Wall Street

2 Shares

About five years ago, I was trying to get a bunch of Big Tech companies to take the right side of an urgent online civil rights fight, and I called an old friend who was very senior at one of the biggest tech companies in the world; they told me that it wasn't going to work, in part because the kinds of people who were coming to tech were there because they wanted to get as rich as possible, no matter what they had to do. My friend contrasted this with earlier eras -- even the dotcom bubble -- when the financial motive was blended with a genuine excitement for the transformative potential of tech to make a fairer, more equitable world. Now, my friend said, the kind of kid who would have gotten an MBA was instead getting an electrical engineering or computer science degree -- not out of any innate love for the subject, but because that was a path to untold riches.

But things are changing. Not only are young people far more skeptical of capitalism and concerned that it will annihilate the human race, but the tech companies' masks have slipped, revealing their willingness to supply ICE and the Chinese government alike, to help the oil industry torch the planet, and to divert their fortunes to supporting white nationalist causes. Companies that tout their ethical center have harbored and even rewarded serial sexual predators and busted nascent union movements.

The tech worker uprisings of recent years have caught the attention of the best and brightest grads of America's elite universities. The New York Times reports that tech recruiters are getting the cold shoulder from students a job fairs, where they are quizzed about Salesforce and Palantir's work with ICE; Google's multimillion-dollar payout for a top exec who engaged in years of demeaning sexual misconduct; Tesla's union-busting; and Facebook's years and years of depraved fuckery (a former Facebook recruiter told the Times that the rate of job-offer acceptances has fallen by 40%!).

Two years ago, I wrote that the tech industry's scarcest resource was tech workers, who are increasingly disillusioned with their employers, and who no longer see themselves as possible CEOs of their own startups, thanks to the monopoly tactics and market concentration that have created a "kill zone" around tech that preclude the possibility of unseating the Big Tech companies with your scrappy startup.

Tech companies can hire new cafeteria workers and gig economy drivers, but talented software engineers, security researchers, and mathematicians are in short supply. They have incredibly leverage over the Big Tech companies, and, as they start to build solidarity with their users and more easily replaced co-workers, they have it within their power to bring Big Tech to its knees.

“It felt like in my freshman year Google, Palantir and Facebook were these shiny places everyone wanted to be. It was like, ‘Wow, you work at Facebook. You must be really smart,’” said Ms. Dogru, 23. “Now if a classmate tells me they’re joining Palantir or Facebook, there’s an awkward gap where they feel like they have to justify themselves.”

Palantir, in particular, has drawn the ire of students at Stanford for providing services to U.S. Immigrations and Customs Enforcement (also known as ICE).

Last summer, a campus activist group, Students for the Liberation of All People, visited the company’s office, a 15-minute walk from campus, and hung a banner nearby that read: “Our software is so powerful it separates families.” Similar protests took place at the University of California, Berkeley, Brown and Yale, according to Recode. The protests, and the attitudes they reflected, were also covered in The Los Angeles Times.

Audrey Steinkamp, a 19-year-old sophomore at Yale, which sends about 10 percent of each graduating class into tech, said that taking a job in Silicon Valley is seen as “selling out,” no different from the economics majors going into consulting who are “lovingly and not-so-lovingly called ‘snakes.’”

‘Techlash’ Hits College Campuses [Emma Goldberg/New York Times]

(Image: No Tech For ICE)

Read the whole story
biocrusoe
201 days ago
reply
Lansing, MI
luizirber
203 days ago
reply
Davis, CA
Share this story
Delete

Fireworks, Long a German New Year’s Eve Tradition, Are Losing Their Luster

1 Share
Faced with the concerns about safety, pollution and potential damage to historic buildings, Germany is rethinking private fireworks.

Read the whole story
biocrusoe
216 days ago
reply
Lansing, MI
Share this story
Delete

Yaks

2 Shares

I’d like to teach scientists how to program, but that would be a lot easier to do if we had better software tools to offer them.

I’d like to teach undergraduates how to design better software tools, but that would be a lot easier to do if they knew more about software engineering.

I’d like to teach undergraduates an evidence-based introduction to software engineering, but that would be a lot easier to do if more people knew how to teach well.

I’d like to teach everyone how to teach well, but that would be a lot easier to do if people in power actually valued and rewarded good teaching.

I’d like to teach people who don’t have power where it comes from and how to take it, but—ah. OK, I guess I’ve found the yak I need to shave.

Read the whole story
biocrusoe
853 days ago
reply
Lansing, MI
luizirber
854 days ago
reply
Davis, CA
Share this story
Delete

Revisiting Wonder Woman

1 Share
It’s been around a year and a half since I first read The Secret History of Wonder Woman and went on a Wonder Woman reading binge.  Despite reading a decent stack of Wonder Woman comics back then, I never really found a story that resonated with me.  My favorite of …

Continue reading

Read the whole story
biocrusoe
1248 days ago
reply
Lansing, MI
Share this story
Delete
Next Page of Stories