Audio Grabber Feature (#1570)

* Creating Audio Grabber

Creating Audio Grabber

Creating Audio Grabber.

Successfully began capturing audio in windows. Starting to implement a hard-coded UV Visualizer.

Got Windows DirectSound Implementation working.
Hardcoded basic VU Meter.

Begin working on linux audio grabber implementation.

Finished Linux Draft Implementation.
Minor Mods to windows implementation.

Windows:
 - Free memory used by device id.
 - Prevent starting audio if the grabber is disabled
 - More debug logging

Linux:
 - Prevent starting audio if the grabber is disabled

Added strings to english
Removed "custom" from device selection
Made hard-coded visualizer values configurable.
wrote values to imageData with BGR priority to enable configurable values to be set in RGB format.
created logic to support "Automatic" to enable the API to select the default device.

Add language key for audio in "Remote Control" section.
Removed audio configuration for number of channels. This was causing an error with some devices.

Fixed logic to update capture while its active.
Optimizing code .

UI Tweaks
Destructuring.

Fixed build error on linux.

Custom Effects - Clean-ups and Enhancements (#1163)

* Cleanup EffectFileHandler

* Support Custom Effect Schemas and align EffectFileHandler

* Change back to colon prefix for system effects

* WebSockets - Fix error in handling fragmented frames

* Correct missing colon updates

* Update json with image file location for custom gif effects

* Image effect deletion - considere full filename is stored in JSON

* Correct selection lists indentions

Creating Audio Grabber

Creating Audio Grabber

Creating Audio Grabber.

Successfully began capturing audio in windows. Starting to implement a hard-coded UV Visualizer.

Got Windows DirectSound Implementation working.
Hardcoded basic VU Meter.

Begin working on linux audio grabber implementation.

Finished Linux Draft Implementation.
Minor Mods to windows implementation.

Windows:
 - Free memory used by device id.
 - Prevent starting audio if the grabber is disabled
 - More debug logging

Linux:
 - Prevent starting audio if the grabber is disabled

Added strings to english
Removed "custom" from device selection
Made hard-coded visualizer values configurable.
wrote values to imageData with BGR priority to enable configurable values to be set in RGB format.
created logic to support "Automatic" to enable the API to select the default device.

Add language key for audio in "Remote Control" section.
Removed audio configuration for number of channels. This was causing an error with some devices.

Fixed logic to update capture while its active.
Optimizing code .

UI Tweaks
Destructuring.

Fixed build error on linux.

Commented setVideoMode from AudioGrabber.

Linux Threading changes.

Implementing new API

Continuing to implement audio into new APIs

Fixed Audio Grabber for DirectSound on Windows
Fixed UI for Audio Grabber Configuration
Default AUDIO to off unless specified.

fixed missing #ifdef for audio grabber.

Added logic to calculate a dynamic multiplier from the signal input.

Updating linux api for discovering devices.

Fixed HTML/JS issues with view.
Fixed NPE in Windows.
Disabled setting thread priority in linux.

updated the schema options check to pass through hidden states and commented the change.

Updated grabber start conditions
Updated Audio grabber to instantiate similar to video grabber

Updated windows grabber to set "started" flag to false when shutting down.
Removed "tryStart" to prevent enabling audio capture unnecessarily.

Fixing instance audio grabber device configuration

Added configurable resolution
Reduced tolerance to 5%
Fixed issue where grabber failed for additional instances when "start" was called multiple times.

Fixed resolution calculation

Change averaging algorithm to prevent overflowing the sum.

Updated logic to stop audio grabber when disabled.

Fix integer casting and rounding.

Restart grabber on configuration change.
Fix missing include/grabber/AudioGrabber.
Disable tolerance.

Added configurable tolerance.
Fixed tolerance algorithm.
reset multiplier on configuration change.

Line Endings

Proposed change and questions/request to fix

implementing more of LordGrey's suggestions.

Fix mode for snd_pcm_open. Latest ALSA uses SND_PCM_NONBLOCK instead of SND_PCM_OPEN_NONBLOCK
defaulted multiplier to 0 "auto"
defaulted tolerance to 20%

changed 100 to 100.0 for pixel value percentage calculation to fix value from being 0.

missed a 100 as a double so precision isn't lost during math operation.

Fix Windows grabber and further cleanups

Enable Audio grabbing in standard build

Remove empty methods

Fix audio capture priority setting

Remove unused code

Clean-up default config

Allow additional json-editor attributes

Allow multiple effects and resetting to defaults

Correct default values

Allow to build for Qt < 5.14

Update CodeQL build dependency

Update build dependencies

Remove effect1 placeholder

* Renamed uvMeter to VU Meter (Volume Unit)
- Fixed issues flagged by code scanning bot.

* Moved stop call into destructor of implementing class.

* Removed commented linux audio channel configuration logic.

---------

Co-authored-by: Michael Rochelle <michael@j2inn.com>
This commit is contained in:
Michael Rochelle
2023-02-19 00:36:39 -08:00
committed by GitHub
parent a1bfa63343
commit acdf733936
50 changed files with 2390 additions and 243 deletions

View File

@@ -58,7 +58,8 @@ $(document).ready(function () {
if (components[idx].name != "ALL") {
if ((components[idx].name === "FORWARDER" && window.currentHyperionInstance != 0) ||
(components[idx].name === "GRABBER" && !window.serverConfig.framegrabber.enable) ||
(components[idx].name === "V4L" && !window.serverConfig.grabberV4L2.enable))
(components[idx].name === "V4L" && !window.serverConfig.grabberV4L2.enable) ||
(components[idx].name === "AUDIO" && !window.serverConfig.grabberAudio.enable))
continue;
var comp_enabled = components[idx].enabled ? "checked" : "";
@@ -104,8 +105,9 @@ $(document).ready(function () {
var screenGrabberAvailable = (window.serverInfo.grabbers.screen.available.length !== 0);
var videoGrabberAvailable = (window.serverInfo.grabbers.video.available.length !== 0);
const audioGrabberAvailable = (window.serverInfo.grabbers.audio.available.length !== 0);
if (screenGrabberAvailable || videoGrabberAvailable) {
if (screenGrabberAvailable || videoGrabberAvailable || audioGrabberAvailable) {
if (screenGrabberAvailable) {
var screenGrabber = window.serverConfig.framegrabber.enable ? $.i18n('general_enabled') : $.i18n('general_disabled');
@@ -120,6 +122,13 @@ $(document).ready(function () {
} else {
$("#dash_video_grabber_row").hide();
}
if (audioGrabberAvailable) {
const audioGrabber = window.serverConfig.grabberAudio.enable ? $.i18n('general_enabled') : $.i18n('general_disabled');
$('#dash_audio_grabber').html(audioGrabber);
} else {
$("#dash_audio_grabber_row").hide();
}
} else {
$("#dash_capture_hw").hide();
}

View File

@@ -4,9 +4,11 @@ $(document).ready(function () {
var screenGrabberAvailable = (window.serverInfo.grabbers.screen.available.length !== 0);
var videoGrabberAvailable = (window.serverInfo.grabbers.video.available.length !== 0);
const audioGrabberAvailable = (window.serverInfo.grabbers.audio.available.length !== 0);
var CEC_ENABLED = (jQuery.inArray("cec", window.serverInfo.services) !== -1);
var conf_editor_video = null;
var conf_editor_audio = null;
var conf_editor_screen = null;
var configuredDevice = "";
@@ -38,6 +40,22 @@ $(document).ready(function () {
}
}
// Audio-Grabber
if (audioGrabberAvailable) {
$('#conf_cont').append(createRow('conf_cont_audio'));
$('#conf_cont_audio').append(createOptPanel('fa-volume', $.i18n("edt_conf_audio_heading_title"), 'editor_container_audiograbber', 'btn_submit_audiograbber', 'panel-system', 'audiograbberPanelId'));
if (storedAccess === 'expert') {
const conf_cont_audio_footer = document.getElementById("editor_container_audiograbber").nextElementSibling;
$(conf_cont_audio_footer).prepend('<button class="btn btn-primary mdi-24px" id="btn_audiograbber_set_effect_defaults" disabled data-toggle="tooltip" data-placement="top" title="' + $.i18n("edt_conf_audio_hardware_set_defaults_tip") + '">'
+ '<i class= "fa fa-fw fa-undo" ></i >' + $.i18n("edt_conf_audio_effect_set_defaults") + '</button > ');
}
if (window.showOptHelp) {
$('#conf_cont_audio').append(createHelpTable(window.schema.grabberAudio.properties, $.i18n("edt_conf_audio_heading_title"), "audiograbberHelpPanelId"));
}
}
JSONEditor.defaults.custom_validators.push(function (schema, value, path) {
var errors = [];
@@ -694,6 +712,121 @@ $(document).ready(function () {
});
}
// External Input Sources (Audio-Grabbers)
if (audioGrabberAvailable) {
conf_editor_audio = createJsonEditor('editor_container_audiograbber', {
grabberAudio: window.schema.grabberAudio
}, true, true);
conf_editor_audio.on('ready', () => {
// Trigger conf_editor_audio.watch - 'root.grabberAudio.enable'
const audioEnable = window.serverConfig.grabberAudio.enable;
conf_editor_audio.getEditor("root.grabberAudio.enable").setValue(audioEnable);
});
conf_editor_audio.on('change', () => {
// Validate the current editor's content
if (!conf_editor_audio.validate().length) {
const deviceSelected = conf_editor_audio.getEditor("root.grabberAudio.available_devices").getValue();
switch (deviceSelected) {
case "SELECT":
showInputOptionsForKey(conf_editor_audio, "grabberAudio", ["enable", "available_devices"], false);
break;
case "NONE":
showInputOptionsForKey(conf_editor_audio, "grabberAudio", ["enable", "available_devices"], false);
break;
default:
window.readOnlyMode ? $('#btn_submit_audiograbber').prop('disabled', true) : $('#btn_submit_audiograbber').prop('disabled', false);
break;
}
}
else {
$('#btn_submit_audiograbber').prop('disabled', true);
}
});
// Enable
conf_editor_audio.watch('root.grabberAudio.enable', () => {
const audioEnable = conf_editor_audio.getEditor("root.grabberAudio.enable").getValue();
if (audioEnable)
{
showInputOptionsForKey(conf_editor_audio, "grabberAudio", "enable", true);
$('#btn_audiograbber_set_effect_defaults').show();
if (window.showOptHelp) {
$('#audiograbberHelpPanelId').show();
}
discoverInputSources("audio");
}
else
{
$('#btn_submit_audiograbber').prop('disabled', false);
$('#btn_audiograbber_set_effect_defaults').hide();
showInputOptionsForKey(conf_editor_audio, "grabberAudio", "enable", false);
$('#audiograbberHelpPanelId').hide();
}
});
// Available Devices
conf_editor_audio.watch('root.grabberAudio.available_devices', () => {
const deviceSelected = conf_editor_audio.getEditor("root.grabberAudio.available_devices").getValue();
if (deviceSelected === "SELECT" || deviceSelected === "NONE" || deviceSelected === "") {
$('#btn_submit_audiograbber').prop('disabled', true);
showInputOptionsForKey(conf_editor_audio, "grabberAudio", ["enable", "available_devices"], false);
}
else
{
showInputOptionsForKey(conf_editor_audio, "grabberAudio", ["enable", "available_devices"], true);
const deviceProperties = getPropertiesOfDevice("audio", deviceSelected);
//Update hidden input element
conf_editor_audio.getEditor("root.grabberAudio.device").setValue(deviceProperties.device);
//Enfore configured JSON-editor dependencies
conf_editor_audio.notifyWatchers("root.grabberAudio.audioEffect");
//Enable set defaults button
$('#btn_audiograbber_set_effect_defaults').prop('disabled', false);
if (conf_editor_audio.validate().length && !window.readOnlyMode) {
$('#btn_submit_audiograbber').prop('disabled', false);
}
}
});
$('#btn_submit_audiograbber').off().on('click', function () {
const saveOptions = conf_editor_audio.getValue();
const instCaptOptions = window.serverConfig.instCapture;
instCaptOptions.audioEnable = true;
saveOptions.instCapture = instCaptOptions;
requestWriteConfig(saveOptions);
});
// ------------------------------------------------------------------
$('#btn_audiograbber_set_effect_defaults').off().on('click', function () {
const currentEffect = conf_editor_audio.getEditor("root.grabberAudio.audioEffect").getValue();
var effectEditor = conf_editor_audio.getEditor("root.grabberAudio." + currentEffect);
var defaultProperties = effectEditor.schema.defaultProperties;
var default_values = {};
for (const item of defaultProperties) {
default_values[item] = effectEditor.schema.properties[item].default;
}
effectEditor.setValue(default_values);
});
}
// ------------------------------------------------------------------
//////////////////////////////////////////////////
@@ -706,6 +839,9 @@ $(document).ready(function () {
if (videoGrabberAvailable) {
createHint("intro", $.i18n('conf_grabber_v4l_intro'), "editor_container_videograbber");
}
if (audioGrabberAvailable) {
createHint("intro", $.i18n('conf_grabber_audio_intro'), "editor_container_audiograbber");
}
}
removeOverlay();
@@ -773,6 +909,38 @@ $(document).ready(function () {
}
};
// build dynamic audio input enum
const updateAudioSourcesList = function (type, discoveryInfo) {
const enumVals = [];
const enumTitelVals = [];
let enumDefaultVal = "";
let addSelect = false;
if (jQuery.isEmptyObject(discoveryInfo)) {
enumVals.push("NONE");
enumTitelVals.push($.i18n('edt_conf_grabber_discovered_none'));
}
else {
for (const device of discoveryInfo) {
enumVals.push(device.device_name);
}
conf_editor_audio.getEditor('root.grabberAudio').enable();
configuredDevice = window.serverConfig.grabberAudio.available_devices;
if ($.inArray(configuredDevice, enumVals) != -1) {
enumDefaultVal = configuredDevice;
}
else {
addSelect = true;
}
}
if (enumVals.length > 0) {
updateJsonEditorSelection(conf_editor_audio, 'root.grabberAudio',
'available_devices', {}, enumVals, enumTitelVals, enumDefaultVal, addSelect, false);
}
};
async function discoverInputSources(type, params) {
const result = await requestInputSourcesDiscovery(type, params);
@@ -782,7 +950,8 @@ $(document).ready(function () {
}
else {
discoveryResult = {
"video_sources": []
"video_sources": [],
"audio_soruces": []
};
}
@@ -799,6 +968,12 @@ $(document).ready(function () {
updateVideoSourcesList(type, discoveredInputSources.video);
}
break;
case "audio":
discoveredInputSources.audio = discoveryResult.audio_sources;
if (audioGrabberAvailable) {
updateAudioSourcesList(type, discoveredInputSources.audio);
}
break;
}
}

View File

@@ -3,6 +3,7 @@ $(document).ready(function () {
var screenGrabberAvailable = (window.serverInfo.grabbers.screen.available.length !== 0);
var videoGrabberAvailable = (window.serverInfo.grabbers.video.available.length !== 0);
const audioGrabberAvailable = (window.serverInfo.grabbers.audio.available.length !== 0);
var BOBLIGHT_ENABLED = (jQuery.inArray("boblight", window.serverInfo.services) !== -1);
@@ -15,7 +16,7 @@ $(document).ready(function () {
// Instance Capture
if (window.showOptHelp) {
if (screenGrabberAvailable || videoGrabberAvailable) {
if (screenGrabberAvailable || videoGrabberAvailable || audioGrabberAvailable) {
$('#conf_cont').append(createRow('conf_cont_instCapt'));
$('#conf_cont_instCapt').append(createOptPanel('fa-camera', $.i18n("edt_conf_instCapture_heading_title"), 'editor_container_instCapt', 'btn_submit_instCapt', ''));
$('#conf_cont_instCapt').append(createHelpTable(window.schema.instCapture.properties, $.i18n("edt_conf_instCapture_heading_title")));
@@ -29,7 +30,7 @@ $(document).ready(function () {
}
else {
$('#conf_cont').addClass('row');
if (screenGrabberAvailable || videoGrabberAvailable) {
if (screenGrabberAvailable || videoGrabberAvailable || audioGrabberAvailable) {
$('#conf_cont').append(createOptPanel('fa-camera', $.i18n("edt_conf_instCapture_heading_title"), 'editor_container_instCapt', 'btn_submit_instCapt', ''));
}
if (BOBLIGHT_ENABLED) {
@@ -37,7 +38,7 @@ $(document).ready(function () {
}
}
if (screenGrabberAvailable || videoGrabberAvailable) {
if (screenGrabberAvailable || videoGrabberAvailable || audioGrabberAvailable) {
// Instance Capture
conf_editor_instCapt = createJsonEditor('editor_container_instCapt', {
@@ -81,12 +82,29 @@ $(document).ready(function () {
showInputOptionForItem(conf_editor_instCapt, "instCapture", "v4lPriority", false);
}
if (audioGrabberAvailable) {
if (!window.serverConfig.grabberAudio.enable) {
conf_editor_instCapt.getEditor("root.instCapture.audioEnable").setValue(false);
conf_editor_instCapt.getEditor("root.instCapture.audioEnable").disable();
}
else {
conf_editor_instCapt.getEditor("root.instCapture.audioEnable").setValue(window.serverConfig.instCapture.audioEnable);
}
} else {
showInputOptionForItem(conf_editor_instCapt, "instCapture", "audioGrabberDevice", false);
showInputOptionForItem(conf_editor_instCapt, "instCapture", "audioEnable", false);
showInputOptionForItem(conf_editor_instCapt, "instCapture", "audioPriority", false);
}
});
conf_editor_instCapt.on('change', function () {
if (!conf_editor_instCapt.validate().length) {
if (!window.serverConfig.framegrabber.enable && !window.serverConfig.grabberV4L2.enable) {
if (!window.serverConfig.framegrabber.enable &&
!window.serverConfig.grabberV4L2.enable &&
!window.serverConfig.grabberAudio.enable) {
$('#btn_submit_instCapt').prop('disabled', true);
} else {
window.readOnlyMode ? $('#btn_submit_instCapt').prop('disabled', true) : $('#btn_submit_instCapt').prop('disabled', false);
@@ -130,6 +148,23 @@ $(document).ready(function () {
}
});
conf_editor_instCapt.watch('root.instCapture.audioEnable', () => {
const audioEnable = conf_editor_instCapt.getEditor("root.instCapture.audioEnable").getValue();
if (audioEnable) {
conf_editor_instCapt.getEditor("root.instCapture.audioGrabberDevice").setValue(window.serverConfig.grabberAudio.available_devices);
conf_editor_instCapt.getEditor("root.instCapture.audioGrabberDevice").disable();
showInputOptions("instCapture", ["audioGrabberDevice"], true);
showInputOptions("instCapture", ["audioPriority"], true);
}
else {
if (!window.serverConfig.grabberAudio.enable) {
conf_editor_instCapt.getEditor("root.instCapture.audioEnable").disable();
}
showInputOptions("instCapture", ["audioGrabberDevice"], false);
showInputOptions("instCapture", ["audioPriority"], false);
}
});
$('#btn_submit_instCapt').off().on('click', function () {
requestWriteConfig(conf_editor_instCapt.getValue());
});

View File

@@ -154,6 +154,9 @@ $(document).ready(function () {
case "V4L":
owner = $.i18n('general_comp_V4L') + ': (' + owner + ')';
break;
case "AUDIO":
owner = $.i18n('general_comp_AUDIO') + ': (' + owner + ')';
break;
case "BOBLIGHTSERVER":
owner = $.i18n('general_comp_BOBLIGHTSERVER');
break;
@@ -219,7 +222,8 @@ $(document).ready(function () {
for (const comp of components) {
if (comp.name === "ALL" || (comp.name === "FORWARDER" && window.currentHyperionInstance != 0) ||
(comp.name === "GRABBER" && !window.serverConfig.framegrabber.enable) ||
(comp.name === "V4L" && !window.serverConfig.grabberV4L2.enable))
(comp.name === "V4L" && !window.serverConfig.grabberV4L2.enable) ||
(comp.name === "AUDIO" && !window.serverConfig.grabberAudio.enable))
continue;
const enable_style = (comp.enabled ? "checked" : "");

View File

@@ -1220,6 +1220,7 @@ function getSystemInfo() {
//info += '- Log lvl: ' + window.serverConfig.logger.level + '\n';
info += '- Avail Screen Cap.: ' + window.serverInfo.grabbers.screen.available + '\n';
info += '- Avail Video Cap.: ' + window.serverInfo.grabbers.video.available + '\n';
info += '- Avail Audio Cap.: ' + window.serverInfo.grabbers.audio.available + '\n';
info += '- Avail Services: ' + window.serverInfo.services + '\n';
info += '- Config path: ' + shy.rootPath + '\n';
info += '- Database: ' + (shy.readOnlyMode ? "ready-only" : "read/write") + '\n';
@@ -1317,9 +1318,9 @@ function showInputOptionsForKey(editor, item, showForKeys, state) {
}
}
for (var key in editor.schema.properties[item].properties) {
for (let key in editor.schema.properties[item].properties) {
if ($.inArray(key, keysToshow) === -1) {
var accessLevel = editor.schema.properties[item].properties[key].access;
const accessLevel = editor.schema.properties[item].properties[key].access;
var hidden = false;
if (editor.schema.properties[item].properties[key].options) {