Audio Grabber Feature (#1570)

* Creating Audio Grabber

Creating Audio Grabber

Creating Audio Grabber.

Successfully began capturing audio in windows. Starting to implement a hard-coded UV Visualizer.

Got Windows DirectSound Implementation working.
Hardcoded basic VU Meter.

Begin working on linux audio grabber implementation.

Finished Linux Draft Implementation.
Minor Mods to windows implementation.

Windows:
 - Free memory used by device id.
 - Prevent starting audio if the grabber is disabled
 - More debug logging

Linux:
 - Prevent starting audio if the grabber is disabled

Added strings to english
Removed "custom" from device selection
Made hard-coded visualizer values configurable.
wrote values to imageData with BGR priority to enable configurable values to be set in RGB format.
created logic to support "Automatic" to enable the API to select the default device.

Add language key for audio in "Remote Control" section.
Removed audio configuration for number of channels. This was causing an error with some devices.

Fixed logic to update capture while its active.
Optimizing code .

UI Tweaks
Destructuring.

Fixed build error on linux.

Custom Effects - Clean-ups and Enhancements (#1163)

* Cleanup EffectFileHandler

* Support Custom Effect Schemas and align EffectFileHandler

* Change back to colon prefix for system effects

* WebSockets - Fix error in handling fragmented frames

* Correct missing colon updates

* Update json with image file location for custom gif effects

* Image effect deletion - considere full filename is stored in JSON

* Correct selection lists indentions

Creating Audio Grabber

Creating Audio Grabber

Creating Audio Grabber.

Successfully began capturing audio in windows. Starting to implement a hard-coded UV Visualizer.

Got Windows DirectSound Implementation working.
Hardcoded basic VU Meter.

Begin working on linux audio grabber implementation.

Finished Linux Draft Implementation.
Minor Mods to windows implementation.

Windows:
 - Free memory used by device id.
 - Prevent starting audio if the grabber is disabled
 - More debug logging

Linux:
 - Prevent starting audio if the grabber is disabled

Added strings to english
Removed "custom" from device selection
Made hard-coded visualizer values configurable.
wrote values to imageData with BGR priority to enable configurable values to be set in RGB format.
created logic to support "Automatic" to enable the API to select the default device.

Add language key for audio in "Remote Control" section.
Removed audio configuration for number of channels. This was causing an error with some devices.

Fixed logic to update capture while its active.
Optimizing code .

UI Tweaks
Destructuring.

Fixed build error on linux.

Commented setVideoMode from AudioGrabber.

Linux Threading changes.

Implementing new API

Continuing to implement audio into new APIs

Fixed Audio Grabber for DirectSound on Windows
Fixed UI for Audio Grabber Configuration
Default AUDIO to off unless specified.

fixed missing #ifdef for audio grabber.

Added logic to calculate a dynamic multiplier from the signal input.

Updating linux api for discovering devices.

Fixed HTML/JS issues with view.
Fixed NPE in Windows.
Disabled setting thread priority in linux.

updated the schema options check to pass through hidden states and commented the change.

Updated grabber start conditions
Updated Audio grabber to instantiate similar to video grabber

Updated windows grabber to set "started" flag to false when shutting down.
Removed "tryStart" to prevent enabling audio capture unnecessarily.

Fixing instance audio grabber device configuration

Added configurable resolution
Reduced tolerance to 5%
Fixed issue where grabber failed for additional instances when "start" was called multiple times.

Fixed resolution calculation

Change averaging algorithm to prevent overflowing the sum.

Updated logic to stop audio grabber when disabled.

Fix integer casting and rounding.

Restart grabber on configuration change.
Fix missing include/grabber/AudioGrabber.
Disable tolerance.

Added configurable tolerance.
Fixed tolerance algorithm.
reset multiplier on configuration change.

Line Endings

Proposed change and questions/request to fix

implementing more of LordGrey's suggestions.

Fix mode for snd_pcm_open. Latest ALSA uses SND_PCM_NONBLOCK instead of SND_PCM_OPEN_NONBLOCK
defaulted multiplier to 0 "auto"
defaulted tolerance to 20%

changed 100 to 100.0 for pixel value percentage calculation to fix value from being 0.

missed a 100 as a double so precision isn't lost during math operation.

Fix Windows grabber and further cleanups

Enable Audio grabbing in standard build

Remove empty methods

Fix audio capture priority setting

Remove unused code

Clean-up default config

Allow additional json-editor attributes

Allow multiple effects and resetting to defaults

Correct default values

Allow to build for Qt < 5.14

Update CodeQL build dependency

Update build dependencies

Remove effect1 placeholder

* Renamed uvMeter to VU Meter (Volume Unit)
- Fixed issues flagged by code scanning bot.

* Moved stop call into destructor of implementing class.

* Removed commented linux audio channel configuration logic.

---------

Co-authored-by: Michael Rochelle <michael@j2inn.com>
This commit is contained in:
Michael Rochelle
2023-02-19 00:36:39 -08:00
committed by GitHub
parent a1bfa63343
commit acdf733936
50 changed files with 2390 additions and 243 deletions

View File

@@ -111,7 +111,7 @@ public:
// server port services
list << "jsonServer" << "protoServer" << "flatbufServer" << "forwarder" << "webConfig" << "network"
// capture
<< "framegrabber" << "grabberV4L2"
<< "framegrabber" << "grabberV4L2" << "grabberAudio"
// other
<< "logger" << "general";

View File

@@ -0,0 +1,196 @@
#ifndef AUDIOGRABBER_H
#define AUDIOGRABBER_H
#include <QObject>
#include <QColor>
#include <cmath>
// Hyperion-utils includes
#include <utils/ColorRgb.h>
#include <hyperion/Grabber.h>
#include <utils/Logger.h>
///
/// Base Audio Grabber Class
///
/// This class is extended by the windows audio grabber to provied DirectX9 access to the audio devices
/// This class is extended by the linux audio grabber to provide ALSA access to the audio devices
///
/// @brief The DirectX9 capture implementation
///
class AudioGrabber : public Grabber
{
Q_OBJECT
public:
///
/// Device properties
///
/// this structure holds the name, id, and inputs of the enumerated audio devices.
///
struct DeviceProperties
{
QString name = QString();
QString id = QString();
QMultiMap<QString, int> inputs = QMultiMap<QString, int>();
};
AudioGrabber();
~AudioGrabber() override;
///
/// Start audio capturing session
///
/// @returns true if successful
virtual bool start();
///
/// Stop audio capturing session
///
virtual void stop();
///
/// Restart the audio capturing session
///
void restart();
Logger* getLog();
///
/// Set Device
///
/// configures the audio device used by the grabber
///
/// @param[in] device identifier of audio device
void setDevice(const QString& device);
///
/// Set Configuration
///
/// sets the audio grabber's configuration parameters
///
/// @param[in] config object of configuration parameters
void setConfiguration(const QJsonObject& config);
///
/// Reset Multiplier
///
/// resets the calcualted audio multiplier so that it is recalculated
/// currently the multiplier is only reduced based on loudness.
///
/// TODO: also calculate a low signal and reset the multiplier
///
void resetMultiplier();
///
/// Discover
///
/// discovers audio devices in the system
///
/// @param[in] params discover parameters
/// @return array of audio devices
virtual QJsonArray discover(const QJsonObject& params);
signals:
void newFrame(const Image<ColorRgb>& image);
protected:
///
/// Process Audio Frame
///
/// this functions takes in an audio buffer and emits a visual representation of the audio data
///
/// @param[in] buffer The audio buffer to process
/// @param[in] length The length of audio data in the buffer
void processAudioFrame(int16_t* buffer, int length);
///
/// Audio device id / properties map
///
/// properties include information such as name, inputs, and etc...
///
QMap<QString, AudioGrabber::DeviceProperties> _deviceProperties;
///
/// Current device
///
QString _device;
///
/// Hot Color
///
/// the color of the leds when the signal is high or hot
///
QColor _hotColor;
///
/// Warn value
///
/// The maximum value of the warning color. above this threshold the signal is considered hot
///
int _warnValue;
///
/// Warn color
///
/// the color of the leds when the signal is in between the safe and warn value threshold
///
QColor _warnColor;
///
/// Save value
///
/// The maximum value of the safe color. above this threshold the signal enteres the warn zone.
/// below the signal is in the safe zone.
///
int _safeValue;
///
/// Safe color
///
/// the color of the leds when the signal is below the safe threshold
///
QColor _safeColor;
///
/// Multiplier
///
/// this value is used to multiply the input signal value. Some inputs may have a very low signal
/// and the multiplier is used to get the desired visualization.
///
/// When the multiplier is configured to 0, the multiplier is automatically configured based off of the average
/// signal amplitude and tolernace.
///
double _multiplier;
///
/// Tolerance
///
/// The tolerance is used to calculate what percentage of the top end part of the signal to ignore when
/// calculating the multiplier. This enables the effect to reach the hot zone with an auto configured multiplier
///
int _tolerance;
///
/// Dynamic Multiplier
///
/// This is the current value of the automatically configured multiplier.
///
double _dynamicMultiplier;
///
/// Started
///
/// true if the capturing session has started.
///
bool _started;
private:
///
/// @brief free the _screen pointer
///
void freeResources();
};
#endif // AUDIOGRABBER_H

View File

@@ -0,0 +1,91 @@
#ifndef AUDIOGRABBERLINUX_H
#define AUDIOGRABBERLINUX_H
#include <pthread.h>
#include <sched.h>
#include <alsa/asoundlib.h>
// Hyperion-utils includes
#include <grabber/AudioGrabber.h>
///
/// @brief The Linux Audio capture implementation
///
class AudioGrabberLinux : public AudioGrabber
{
public:
AudioGrabberLinux();
~AudioGrabberLinux() override;
///
/// Process audio buffer
///
void processAudioBuffer(snd_pcm_sframes_t frames);
///
/// Is Running Flag
///
std::atomic<bool> _isRunning;
///
/// Current capture device
///
snd_pcm_t * _captureDevice;
public slots:
///
/// Start audio capturing session
///
/// @returns true if successful
bool start() override;
///
/// Stop audio capturing session
///
void stop() override;
///
/// Discovery audio devices
///
QJsonArray discover(const QJsonObject& params) override;
private:
///
/// Refresh audio devices
///
void refreshDevices();
///
/// Configure current audio capture interface
///
bool configureCaptureInterface();
///
/// Get device name from path
///
QString getDeviceName(const QString& devicePath) const;
///
/// Current sample rate
///
unsigned int _sampleRate;
///
/// Audio capture thread
///
pthread_t _audioThread;
///
/// ALSA device configuration parameters
///
snd_pcm_hw_params_t * _captureDeviceConfig;
};
///
/// Audio processing thread function
///
static void* AudioThreadRunner(void* params);
#endif // AUDIOGRABBERLINUX_H

View File

@@ -0,0 +1,81 @@
#ifndef AUDIOGRABBERWINDOWS_H
#define AUDIOGRABBERWINDOWS_H
// Hyperion-utils includes
#include <grabber/AudioGrabber.h>
#include <DSound.h>
///
/// @brief The Windows Audio capture implementation
///
class AudioGrabberWindows : public AudioGrabber
{
public:
AudioGrabberWindows();
~AudioGrabberWindows() override;
public slots:
bool start() override;
void stop() override;
QJsonArray discover(const QJsonObject& params) override;
private:
void refreshDevices();
bool configureCaptureInterface();
QString getDeviceName(const QString& devicePath) const;
void processAudioBuffer();
LPDIRECTSOUNDCAPTURE8 recordingDevice;
LPDIRECTSOUNDCAPTUREBUFFER8 recordingBuffer;
HANDLE audioThread;
DWORD bufferCapturePosition;
DWORD bufferCaptureSize;
DWORD notificationSize;
static DWORD WINAPI AudioThreadRunner(LPVOID param);
HANDLE notificationEvent;
std::atomic<bool> isRunning{ false };
static BOOL CALLBACK DirectSoundEnumProcessor(LPGUID deviceIdGuid, LPCTSTR deviceDescStr,
LPCTSTR deviceModelStr, LPVOID context)
{
// Skip undefined audio devices
if (deviceIdGuid == NULL)
return TRUE;
QMap<QString, AudioGrabber::DeviceProperties>* devices = (QMap<QString, AudioGrabber::DeviceProperties>*)context;
AudioGrabber::DeviceProperties device;
// Process Device ID
LPOLESTR deviceIdStr;
HRESULT res = StringFromCLSID(*deviceIdGuid, &deviceIdStr);
if (FAILED(res))
{
Error(Logger::getInstance("AUDIOGRABBER"), "Failed to get CLSID-string for %s with error: 0x%08x: %s", deviceDescStr, res, std::system_category().message(res).c_str());
return FALSE;
}
QString deviceId = QString::fromWCharArray(deviceIdStr);
CoTaskMemFree(deviceIdStr);
// Process Device Information
QString deviceName = QString::fromLocal8Bit(deviceDescStr);
Debug(Logger::getInstance("AUDIOGRABBER"), "Found Audio Device: %s", deviceDescStr);
device.id = deviceId;
device.name = deviceName;
devices->insert(deviceId, device);
return TRUE;
}
};
#endif // AUDIOGRABBERWINDOWS_H

View File

@@ -0,0 +1,69 @@
#pragma once
#include <hyperion/GrabberWrapper.h>
#ifdef WIN32
#include <grabber/AudioGrabberWindows.h>
#endif
#ifdef __linux__
#include <grabber/AudioGrabberLinux.h>
#endif
///
/// Audio Grabber wrapper
///
class AudioWrapper : public GrabberWrapper
{
public:
// The AudioWrapper has no params...
///
/// Constructs the Audio grabber with a specified grab size and update rate.
///
/// @param[in] device Audio Device Identifier
/// @param[in] updateRate_Hz The audio grab rate [Hz]
///
AudioWrapper();
///
/// Destructor of this Audio grabber. Releases any claimed resources.
///
~AudioWrapper() override;
///
/// Settings update handler
///
void handleSettingsUpdate(settings::type type, const QJsonDocument& config) override;
public slots:
///
/// Performs a single frame grab and computes the led-colors
///
void action() override;
///
/// Start audio capturing session
///
/// @returns true if successful
bool start() override;
///
/// Stop audio capturing session
///
void stop() override;
private:
void newFrame(const Image<ColorRgb>& image);
/// The actual grabber
#ifdef WIN32
AudioGrabberWindows _grabber;
#endif
#ifdef __linux__
AudioGrabberLinux _grabber;
#endif
};

View File

@@ -4,12 +4,14 @@
enum class GrabberType {
SCREEN,
VIDEO,
AUDIO,
};
enum class GrabberTypeFilter {
ALL,
SCREEN,
VIDEO,
AUDIO,
};
#endif // GRABBERTYPE_H

View File

@@ -20,6 +20,7 @@ public:
void setSystemCaptureEnable(bool enable);
void setV4LCaptureEnable(bool enable);
void setAudioCaptureEnable(bool enable);
private slots:
///
@@ -48,11 +49,22 @@ private slots:
///
void handleV4lImage(const QString& name, const Image<ColorRgb> & image);
///
/// @brief forward audio image
/// @param image The image
///
void handleAudioImage(const QString& name, const Image<ColorRgb>& image);
///
/// @brief Is called from _v4lInactiveTimer to set source after specific time to inactive
///
void setV4lInactive();
///
/// @brief Is called from _audioInactiveTimer to set source after specific time to inactive
///
void setAudioInactive();
///
/// @brief Is called from _systemInactiveTimer to set source after specific time to inactive
///
@@ -73,4 +85,10 @@ private:
quint8 _v4lCaptPrio;
QString _v4lCaptName;
QTimer* _v4lInactiveTimer;
/// Reflect state of audio capture and prio
bool _audioCaptEnabled;
quint8 _audioCaptPrio;
QString _audioCaptName;
QTimer* _audioInactiveTimer;
};

View File

@@ -43,8 +43,10 @@ public:
static QMap<int, QString> GRABBER_SYS_CLIENTS;
static QMap<int, QString> GRABBER_V4L_CLIENTS;
static QMap<int, QString> GRABBER_AUDIO_CLIENTS;
static bool GLOBAL_GRABBER_SYS_ENABLE;
static bool GLOBAL_GRABBER_V4L_ENABLE;
static bool GLOBAL_GRABBER_AUDIO_ENABLE;
///
/// Starts the grabber which produces led values with the specified update rate
@@ -78,6 +80,8 @@ public:
void setSysGrabberState(bool sysGrabberState){ GLOBAL_GRABBER_SYS_ENABLE = sysGrabberState; }
bool getV4lGrabberState() const { return GLOBAL_GRABBER_V4L_ENABLE; }
void setV4lGrabberState(bool v4lGrabberState){ GLOBAL_GRABBER_V4L_ENABLE = v4lGrabberState; }
bool getAudioGrabberState() const { return GLOBAL_GRABBER_AUDIO_ENABLE; }
void setAudioGrabberState(bool audioGrabberState) { GLOBAL_GRABBER_AUDIO_ENABLE = audioGrabberState; }
static QStringList availableGrabbers(GrabberTypeFilter type = GrabberTypeFilter::ALL);

View File

@@ -454,6 +454,9 @@ signals:
/// Signal which is emitted, when a new V4l proto image should be forwarded
void forwardV4lProtoMessage(const QString&, const Image<ColorRgb>&);
/// Signal which is emitted, when a new Audio proto image should be forwarded
void forwardAudioProtoMessage(const QString&, const Image<ColorRgb>&);
#if defined(ENABLE_FLATBUF_SERVER) || defined(ENABLE_PROTOBUF_SERVER)
/// Signal which is emitted, when a new Flat-/Proto- Buffer image should be forwarded
void forwardBufferMessage(const QString&, const Image<ColorRgb>&);

View File

@@ -23,6 +23,7 @@ enum Components
#endif
COMP_GRABBER,
COMP_V4L,
COMP_AUDIO,
COMP_COLOR,
COMP_IMAGE,
COMP_EFFECT,
@@ -50,6 +51,7 @@ inline const char* componentToString(Components c)
#endif
case COMP_GRABBER: return "Framegrabber";
case COMP_V4L: return "V4L capture device";
case COMP_AUDIO: return "Audio capture device";
case COMP_COLOR: return "Solid color";
case COMP_EFFECT: return "Effect";
case COMP_IMAGE: return "Image";
@@ -79,6 +81,7 @@ inline const char* componentToIdString(Components c)
#endif
case COMP_GRABBER: return "GRABBER";
case COMP_V4L: return "V4L";
case COMP_AUDIO: return "AUDIO";
case COMP_COLOR: return "COLOR";
case COMP_EFFECT: return "EFFECT";
case COMP_IMAGE: return "IMAGE";
@@ -107,6 +110,7 @@ inline Components stringToComponent(const QString& component)
#endif
if (cmp == "GRABBER") return COMP_GRABBER;
if (cmp == "V4L") return COMP_V4L;
if (cmp == "AUDIO") return COMP_AUDIO;
if (cmp == "COLOR") return COMP_COLOR;
if (cmp == "EFFECT") return COMP_EFFECT;
if (cmp == "IMAGE") return COMP_IMAGE;

View File

@@ -56,6 +56,13 @@ signals:
void setBufferImage(const QString& name, const Image<ColorRgb>& image);
#endif
///
/// @brief PIPE audioCapture images from audioCapture over HyperionDaemon to Hyperion class
/// @param name The name of the audio capture (path) that is currently active
/// @param image The prepared image
///
void setAudioImage(const QString& name, const Image<ColorRgb>& image);
///
/// @brief PIPE the register command for a new global input over HyperionDaemon to Hyperion class
/// @param[in] priority The priority of the channel

View File

@@ -19,6 +19,7 @@ namespace settings {
SYSTEMCAPTURE,
GENERAL,
V4L2,
AUDIO,
JSONSERVER,
LEDCONFIG,
LEDS,
@@ -52,6 +53,7 @@ namespace settings {
case SYSTEMCAPTURE: return "framegrabber";
case GENERAL: return "general";
case V4L2: return "grabberV4L2";
case AUDIO: return "grabberAudio";
case JSONSERVER: return "jsonServer";
case LEDCONFIG: return "ledConfig";
case LEDS: return "leds";
@@ -84,6 +86,7 @@ namespace settings {
else if (type == "framegrabber") return SYSTEMCAPTURE;
else if (type == "general") return GENERAL;
else if (type == "grabberV4L2") return V4L2;
else if (type == "grabberAudio") return AUDIO;
else if (type == "jsonServer") return JSONSERVER;
else if (type == "ledConfig") return LEDCONFIG;
else if (type == "leds") return LEDS;