A11y Lab @ Penn State

About A11y

A11y Lab is a research group in Penn State's College of Information Sciences and Technology. It works on human-computer interaction, specifically accessible computing — making technology work well for people across a wide range of abilities.

The lab's approach combines human-AI teaming with inclusive design, building tools that fit the people using them rather than the other way around.

What I worked on

I built an Android app that records motion data from a Movesense IMU sensor. This was the starting point for the rest of the project.

I added a Bluetooth screen that remembers previously paired sensors and scans for new ones, so getting connected takes a couple of taps instead of a trip through system settings.

I ran a data collection session with a participant with special needs, and adjusted how we recorded based on what came out of it.

I compared several machine learning algorithms to see which one detected gestures most accurately.

It became clear we needed video alongside the sensor data to label it properly, so I added camera recording to the app.

I started a Python tool for labelling and viewing the recorded data. In early testing it cut processing time by at least 80%.

Where it was headed

The labelling and visualization tools were built and in testing when I left.

The next step was collecting data from a wider group of participants, so the dataset covered more kinds of people.

After that, the plan was to fold the labelling tools into the Android app itself, so people could record and process their own data and tune the model to themselves.