Source-linked AI summary
Liquified protein vibrations, classification and cross-paradigm de novo image generation using deep neural networks
Markus J. Buehler
TL;DR
Protein molecular vibrations have been sonified, and this paper explores a complementary way to render them as macroscopic water-surface images. The authors train neural networks on those images and report classification of tested proteins, including cases with minute spectral differences, alongside DeepDream visualizations. The method is limited to structures represented in training and may depend on experimental setup geometry.
Problem
The paper explores how protein molecular vibrations can be made visible as macroscopic images and analyzed with neural networks.
Source: §1 · pp. 1–2Method
The authors convert protein vibrational spectra into water-film surface waves, train neural networks on the resulting images, and use DeepDream to accentuate learned features.
Source: §2 · p. 2, §2.6 · p. 3Results
Both models correctly classified all tested labels from new surface-wave images, and distinguished proteins with minute vibrational-spectrum differences.
Source: §3 · p. 4Takeaways & Limitations
The method provides a way to visualize molecular vibrations and explore their image features, including in photographs beyond the experimental water-surface images.
Source: §3 · p. 4, §4 · p. 5The method can classify only protein structures represented in its training, and the images may depend on experimental setup geometry.
Source: §4 · p. 5
Abstract
from arXiv · showhide
In recent work we reported the vibrational spectrum of more than 100,000 known protein structures, and a self-consistent sonification method to render the spectrum in the audible range of frequencies (Extreme Mechanics Letters, 2019). Here we present a method to transform these molecular vibrations into materialized vibrations of thin water films using acoustic actuators, leading to complex patterns of surface waves, and using the resulting macroscopic images in further processing using deep convolutional neural networks. Specifically, the patterns of water surface waves for each protein structure is used to build training sets for neural networks, aimed to classify and further process the patterns. Once trained, the neural network model is capable of discerning different proteins solely by analyzing the macroscopic surface wave patterns in the water film. Not only can the method distinguish different types of proteins (e.g. alpha-helix vs hybrids of alpha-helices and beta-sheets), but it is also capable of determining different folding states of the same protein, or the binding events of proteins to ligands. Using the DeepDream algorithm, instances of key features of the deep neural network can be made visible in a range of images, allowing us to explore the inner workings of protein surface wave patter neural networks, as well as the creation of new images by finding and highlighting features of protein molecular spectra in a range of photographic input. The integration of the water-focused realization of cymatics, combined with neural networks and especially generative methods, offer a new direction to realize materiomusical "Inceptionism" as a possible direction in nano-inspired art. The method could have applications for detecting different protein structures, the effect of mutations, or uses in medical imaging and diagnostics, with broad impact in nano-to-macro transitions.
1. Introduction
Protein vibrations can be rendered as audible signals, and this work presents a complementary way to materialize them as visible surface-wave patterns in thin water films. The resulting images are used to train a classifier to identify protein structures from surface waves.
1. Introduction
- Molecular vibrations are normal-mode motion, and prior work reported spectra for more than 100,000 protein structures and a method for sonifying them.The earlier sonification work also informed generative neural networks for de novo protein design.Source: §1 · p. 1
- Acoustic actuators translate protein molecular vibrations into surface waves in a thin water film, producing visible macroscopic patterns.The approach follows Chladni's concept and is similar to cymatics.Source: §1 · p. 1
- Artificial intelligence is used to classify and augment sound-generated images, including by highlighting features detected in deep-network layers.The authors frame this as a way to render sound visible and explore image augmentation.Source: §1 · p. 2
- The study uses images of those patterns to train a model that identifies protein structure from surface waves alone.The workflow calculates and sonifies molecular spectra, excites the water film, collects images, and validates classifier predictions.Source: §1 · p. 2
2. Materials and Methods
The methods convert protein vibrational spectra into water-surface images, then use neural networks for classification and DeepDream visualization. The study describes the experimental setup, image preparation, model alternatives, and use of protein structure identifiers.
2. Materials and Methods
- The protein synthesizer generates audible signals from molecular vibrational spectra, which are fed to an actuator attached to a water-filled petri dish.The synthesizer was published in earlier work; pure sine waves were also generated for comparison.Source: §2 · p. 2, §2.3 · p. 2
2.1 Protein structure labeling
- Protein structures are identified by their Protein Data Bank (PDB) codes.The codes can be downloaded and explored through the RCSB website.Source: §2.1 · p. 2
2.2 Experimental Setup
- An actuator mounted to a fixed, high-mass substrate vibrates a petri dish containing a 1 mm water layer, which is imaged under fixed camera conditions.The dish was spray-painted white for optical contrast, and the water depth was selected to produce surface waves within available actuation energy.Source: §2.2 · p. 2
2.3 Sound generation
- Audio signals are generated with Ableton Live and Max/MSP using implementations of the protein synthesizer, alongside pure sine-wave signals.The protein synthesizer was described in earlier work.Source: §2.3 · p. 2
2.4 Image capturing and pre-processing
- Video is recorded at 60 fps and 1920x1080, then cropped to 585x788 frames without additional image processing; 80% of images train the model and 20% validate it.New images from additional experimental imaging are used to test predictions.Source: §2.4 · p. 3
2.5 Neural network design and training
- The study compares a fine-tuned 50-layer pretrained ResNet with a custom 20-layer convolutional and pooling network.The ResNet has 23,587,789 parameters, while the custom network has 945,165.Source: §2.5 · p. 3
2.6 DeepDream generative method
- DeepDream generates images by adjusting an input image to accentuate patterns associated with activations in selected network layers.Different layers, including convolutional and pooling layers, are examined to make learned features visible.Source: §2.6 · p. 3
3. Results
Both neural-network models classify the tested surface-wave images, while DeepDream visualizes learned features in experimental and other images. The results include discrimination of protein structures with minute spectral differences and layer-dependent visualizations.
3. Results
- The pretrained ResNet converges faster and more stably than the custom network during training.The authors attribute this to its pretrained parameters.Source: §3 · p. 4
- Both models correctly predict all 13 tested labels from new surface-wave images, with the correct-class score close to 1.00.The classification test averages scores across new images, and the authors report prediction without error.Source: §3 · p. 4
- Both models distinguish protein structures across the studied cases, including 6m17 and 6m18 despite their minute vibrational-spectrum differences.The results also include structures ranging from a small protein to a very large protein.Source: §3 · p. 4
- DeepDream accentuates recognized patterns in water-surface images and captures relevant features in other photographs and landscapes, though the fit is weaker for those other images.The method is presented as a way to create generative art from molecular vibrations.Source: §3 · p. 4
- Selected internal network layers visualize surface-wave features in other image types, while a conventional Inception model displays features learned from conventional images.The Inception model was not trained on the study's surface-wave dataset.Source: §3 · pp. 4–5
4. Discussion and conclusion
The paper presents a water-film approach for visualizing protein vibrations and reports that its classifier detects subtle spectral and protein-state differences. Its scope is limited to trained structures, and the resulting patterns may depend on experimental geometry and settings.
4. Discussion and conclusion
- The method offers a first way to visualize protein molecular vibrations and integrate water-focused cymatics with artificial intelligence and generative methods.The authors frame materiomusical “Inceptionism” as a possible future direction in nano-inspired art.Source: §4 · p. 5
- Minute spectral changes between 6m17 and 6m18, and different states of 6vxx and 6vsb, can be detected with the method.The authors note that building visual representations of all known proteins remains beyond this study's scope.Source: §4 · p. 5
- Classification is limited to protein structures used to train the neural network, and surface-wave patterns may depend on water level, container size, and actuator power.The authors identify experimental-setup dependence as another limitation.Source: §4 · p. 5
Figures and captions
The figures trace the workflow from protein spectra to water-wave images and classification, then show tests of the models and DeepDream visualizations. They also compare protein cases and image-processing approaches.
Figures and captions
- Figure 1A traces molecular spectra through water-film actuation and image-based classification, while Figure 1B surveys proteins from simple helices to complex structures.The examples include COVID-19 spike proteins bound and unbound to ACE2.Source: Figure 1 · p. 9
- Figures 2–3 show actuator-driven water waves being captured and cropped, with panel C identifying the images used for neural-network training.Figure 3 contrasts reflective and white backgrounds before showing cropped training inputs.Source: Figure 2 · p. 10, Figure 3 · p. 11
- Figures 4–5 present a 20-layer, 945,165-parameter custom network and training results for it and a transfer-learned ResNet.The caption compares the custom model with ResNet's 23,587,789 parameters and 50 layers.Source: Figure 4 · p. 12, Figure 5 · p. 13
- Figure 6 reports averaged radar scores on new images; both models assign the highest score to the correct label across the cases studied.The test cases include proteins, sine waves at different frequencies, and silence.Source: Figure 6 · p. 14
- Figure 7 compares normal-mode frequencies and audio spectrograms for bound 6m17 and unbound 6m18, which the model distinguishes despite small spectral differences.Panel A plots frequencies over mode number; panel B shows the resulting audio spectrogram.Source: Figure 7 · p. 15
- Figures 8–10 use selected network layers to visualize learned features in processed images, including features captured at different scales.Figure 8 shows original and processed images; Figures 9–10 collect examples from internal layers.Source: Figure 8 · p. 16, Figure 9 · p. 17, Figure 10 · p. 18
- Figure 11 applies the untrained-on-this-dataset Inception model to images, whose patterns reflect features learned by a conventional classifier.The model is used as-is rather than trained against the paper's dataset.Source: Figure 11 · p. 19