Donate to e Foundation | Murena handsets with /e/OS | Own a part of Murena! Learn more

Commit 6be8f3ea authored by Lev Proleev's avatar Lev Proleev Committed by Slava Shklyaev
Browse files

Add parallel linking support for BIDIRECTIONAL_SEQUENCE_{LSTM|RNN} 

Change-Id: I51396f82de8dd7282db85a26b0a1be84b236b8c1
Merged-In: I51396f82de8dd7282db85a26b0a1be84b236b8c1
Fix: 138653129
Test: NNTest_static and VtsHalNeuralnetworksV1_3TargetTest
(cherry picked from commit 13c43c00)
parent d7975bf8

current.txt

+2 −2
Original line number Diff line number Diff line
@@ -578,7 +578,7 @@ eb2fa0c883c2185d514be0b84c179b283753ef0c1b77b45b4f359bd23bba8b75 android.hardwar 
5f6d3097ba84cb63c430787123f4de1b31c11f90b531b98eae9a8623a5ae962a android.hardware.neuralnetworks@1.1::types

fb382e986c10b8fbb797a8546e8f9ea6d1107bfe6f3fb7e57f6bbbf1f807a906 android.hardware.neuralnetworks@1.2::IDevice

40e71cd693de5b832325c5d8f081f2ff20a7ba2b89d401cee5b4b3eb0e241681 android.hardware.neuralnetworks@1.2::IPreparedModel

6c29d6fdd5445911df5456b3b84b949cdd59fca0c0b5507662f26a5cac0cf5e5 android.hardware.neuralnetworks@1.2::types

00649d29680f2c47edf60000c3ae7ae906ba638f0616947147e3676a83cf36fa android.hardware.neuralnetworks@1.2::types

a785a57447a81e9c130eef6904c3a5c256076c6a04588c40620ebd6fa2660d77 android.hardware.radio@1.2::types

1a6e2bd289f22931c526b21916910f1d4c436b7acb9556e4243de4ce8e6cc2e4 android.hardware.soundtrigger@2.0::ISoundTriggerHwCallback

fd65298e1e09e0e3c781ab18305920d757dbe55a3b459ce17814ec5cf6dfee99 android.hardware.wifi@1.0::IWifiP2pIface
@@ -630,7 +630,7 @@ ddcf89cd8ee2df0d32aee55050826446fb64f7aafde0a7cd946c64f61b1a364c android.hardwar 
2fa3679ad7c94b5e88724adcd560c561041068a4ca565c63830e68101988746a android.hardware.neuralnetworks@1.3::IFencedExecutionCallback

43088ffc71945b463a7279262cfe2e290f6ed2f15d3fd6032798a3be299fb08f android.hardware.neuralnetworks@1.3::IPreparedModel

0439a1fbbec7f16e5e4c653d85ac685d51bfafbae15b8f8cca530acdd7d6a8ce android.hardware.neuralnetworks@1.3::IPreparedModelCallback

306fda32ac969fd51d75d066352cadcb769944ec4823be4cdd3f86fdb9e97511 android.hardware.neuralnetworks@1.3::types

dd39887aa4fb60ce60ea9cc043edeadbbae6e922d09d3946311b0b410024ae14 android.hardware.neuralnetworks@1.3::types

3e01d4446cd69fd1c48f8572efd97487bc179564b32bd795800b97bbe10be37b android.hardware.wifi@1.4::IWifi

a64467bae843569f0d465c5be7f0c7a5b987985b55a3ef4794dd5afc68538650 android.hardware.wifi.supplicant@1.3::ISupplicant

44445b8a03d7b9e68b2fbd954672c18a8fce9e32851b0692f4f4ab3407f86ecb android.hardware.wifi.supplicant@1.3::ISupplicantStaIface

neuralnetworks/1.2/types.hal

+40 −9
Original line number Diff line number Diff line
@@ -2314,7 +2314,38 @@ enum OperationType : int32_t { 
    AXIS_ALIGNED_BBOX_TRANSFORM = 41,



    /**

     * Performs a forward LSTM on the input followed by a backward LSTM.

     * A recurrent neural network layer that applies an LSTM cell to a

     * sequence of inputs in forward and backward directions.

     *

     * The op supports cross-linking via an auxiliary input. Regular cell feeds

     * one input into the two RNN cells in the following way:

     *

     *       INPUT  (INPUT_REVERSED)

     *         |         |

     *    ---------------------

     *    | FW_LSTM   BW_LSTM |

     *    ---------------------

     *         |         |

     *      FW_OUT     BW_OUT

     *

     * An op with cross-linking takes two inputs and feeds them into the RNN

     * cells in the following way:

     *

     *       AUX_INPUT   (AUX_INPUT_REVERSED)

     *           |             |

     *     INPUT | (INPUT_R'D.)|

     *       |   |       |     |

     *    -----------------------

     *    |  \  /        \    / |

     *    | FW_LSTM     BW_LSTM |

     *    -----------------------

     *         |           |

     *      FW_OUT      BW_OUT

     *

     * The cross-linking mode is enabled iff auxiliary input and auxiliary

     * weights are present. While stacking this op on top of itself, this

     * allows to connect both forward and backward outputs from previous cell

     * to the next cell's input.

     *

     * Supported tensor {@link OperandType}:

     * * {@link OperandType::TENSOR_FLOAT16}
@@ -2324,7 +2355,6 @@ enum OperationType : int32_t { 
     *

     * All input and output tensors must be of the same type.

     *

     *

     * Inputs:

     * * 0: The input.

     *      A 3-D tensor of shape:
@@ -2533,8 +2563,8 @@ enum OperationType : int32_t { 
     * * “activation” is the function passed as the “fused_activation_function”

     *   argument (if not “NONE”).

     *

     * The op also supports an auxiliary input. Regular cell feeds one input

     * into the two RNN cells in the following way:

     * The op supports cross-linking via an auxiliary input. Regular cell feeds

     * one input into the two RNN cells in the following way:

     *

     *       INPUT  (INPUT_REVERSED)

     *         |         |
@@ -2544,8 +2574,8 @@ enum OperationType : int32_t { 
     *         |         |

     *      FW_OUT     BW_OUT

     *

     * An op with an auxiliary input takes two inputs and feeds them into the

     * RNN cells in the following way:

     * An op with cross-linking takes two inputs and feeds them into the RNN

     * cells in the following way:

     *

     *       AUX_INPUT   (AUX_INPUT_REVERSED)

     *           |             |
@@ -2558,9 +2588,10 @@ enum OperationType : int32_t { 
     *         |           |

     *      FW_OUT      BW_OUT

     *

     * While stacking this op on top of itself, this allows to connect both

     * forward and backward outputs from previous cell to the next cell's

     * inputs.

     * The cross-linking mode is enabled iff auxiliary input and auxiliary

     * weights are present. While stacking this op on top of itself, this

     * allows to connect both forward and backward outputs from previous cell

     * to the next cell's input.

     *

     * Supported tensor {@link OperandType}:

     * * {@link OperandType::TENSOR_FLOAT16}

neuralnetworks/1.3/types.hal

+107 −29
Original line number Diff line number Diff line
@@ -2364,7 +2364,54 @@ enum OperationType : int32_t { 
    AXIS_ALIGNED_BBOX_TRANSFORM = @1.2::OperationType:AXIS_ALIGNED_BBOX_TRANSFORM,



    /**

     * Performs a forward LSTM on the input followed by a backward LSTM.

     * A recurrent neural network layer that applies an LSTM cell to a

     * sequence of inputs in forward and backward directions.

     *

     * The op supports cross-linking via an auxiliary input. Regular cell feeds

     * one input into the two RNN cells in the following way:

     *

     *       INPUT  (INPUT_REVERSED)

     *         |         |

     *    ---------------------

     *    | FW_LSTM   BW_LSTM |

     *    ---------------------

     *         |         |

     *      FW_OUT     BW_OUT

     *

     * An op with cross-linking takes two inputs and feeds them into the RNN

     * cells in the following way:

     *

     *       AUX_INPUT   (AUX_INPUT_REVERSED)

     *           |             |

     *     INPUT | (INPUT_R'D.)|

     *       |   |       |     |

     *    -----------------------

     *    |  \  /        \    / |

     *    | FW_LSTM     BW_LSTM |

     *    -----------------------

     *         |           |

     *      FW_OUT      BW_OUT

     *

     * The cross-linking mode is enabled iff auxiliary input and auxiliary

     * weights are present. While stacking this op on top of itself, this

     * allows to connect both forward and backward outputs from previous cell

     * to the next cell's input.

     *

     * Since HAL version 1.3 parallel linking mode is supported. The mode is

     * enabled if auxiliary input is present but auxiliary weights are omitted.

     * In this case, the cell feeds inputs into the RNN in the following way:

     *

     *       INPUT (AUX_INPUT_REVERSED)

     *         |         |

     *    ---------------------

     *    | FW_LSTM   BW_LSTM |

     *    ---------------------

     *         |         |

     *      FW_OUT     BW_OUT

     *

     * While stacking this op on top of itself, this allows to connect both

     * forward and backward outputs from previous cell to the next cell's

     * corresponding inputs.

     *

     * Supported tensor {@link OperandType}:

     * * {@link OperandType::TENSOR_FLOAT16}
@@ -2374,7 +2421,6 @@ enum OperationType : int32_t { 
     *

     * All input and output tensors must be of the same type.

     *

     *

     * Inputs:

     * * 0: The input.

     *      A 3-D tensor of shape:
@@ -2466,25 +2512,34 @@ enum OperationType : int32_t { 
     * * 38: The backward input cell state.

     *       A 2-D tensor of shape [batch_size, bw_num_units].

     * * 39: The auxiliary input. Optional.

     *       A 3-D tensor of shape [max_time, batch_size, input_size], where “batch_size”

     *       corresponds to the batching dimension, and “input_size” is the size

     *       of the input.

     * * 40: The forward auxiliary input-to-input weights. Optional.

     *       A 2-D tensor of shape [fw_num_units, input_size].

     * * 41: The forward auxiliary input-to-forget weights. Optional.

     *       A 2-D tensor of shape [fw_num_units, input_size].

     * * 42: The forward auxiliary input-to-cell weights. Optional.

     *       A 2-D tensor of shape [fw_num_units, input_size].

     * * 43: The forward auxiliary input-to-output weights. Optional.

     *       A 2-D tensor of shape [fw_num_units, input_size].

     * * 44: The backward auxiliary input-to-input weights. Optional.

     *       A 2-D tensor of shape [bw_num_units, input_size].

     * * 45: The backward auxiliary input-to-forget weights. Optional.

     *       A 2-D tensor of shape [bw_num_units, input_size].

     * * 46: The backward auxiliary input-to-cell weights. Optional.

     *       A 2-D tensor of shape [bw_num_units, input_size].

     * * 47: The backward auxiliary input-to-output weights. Optional.

     *       A 2-D tensor of shape [bw_num_units, input_size].

     *       A 3-D tensor of shape [max_time, batch_size, aux_input_size],

     *       where “batch_size” corresponds to the batching dimension, and

     *       “aux_input_size” is the size of the auxiliary input. Optional. See

     *       the docs above for the usage modes explanation.

     * * 40: The forward auxiliary input-to-input weights.

     *       Optional. See the docs above for the usage modes explanation.

     *       A 2-D tensor of shape [fw_num_units, aux_input_size].

     * * 41: The forward auxiliary input-to-forget weights.

     *       Optional. See the docs above for the usage modes explanation.

     *       A 2-D tensor of shape [fw_num_units, aux_input_size].

     * * 42: The forward auxiliary input-to-cell weights.

     *       Optional. See the docs above for the usage modes explanation.

     *       A 2-D tensor of shape [fw_num_units, aux_input_size].

     * * 43: The forward auxiliary input-to-output weights.

     *       Optional. See the docs above for the usage modes explanation.

     *       A 2-D tensor of shape [fw_num_units, aux_input_size].

     * * 44: The backward auxiliary input-to-input weights.

     *       Optional. See the docs above for the usage modes explanation.

     *       A 2-D tensor of shape [bw_num_units, aux_input_size].

     * * 45: The backward auxiliary input-to-forget weights.

     *       Optional. See the docs above for the usage modes explanation.

     *       A 2-D tensor of shape [bw_num_units, aux_input_size].

     * * 46: The backward auxiliary input-to-cell weights.

     *       Optional. See the docs above for the usage modes explanation.

     *       A 2-D tensor of shape [bw_num_units, aux_input_size].

     * * 47: The backward auxiliary input-to-output weights.

     *       Optional. See the docs above for the usage modes explanation.

     *       A 2-D tensor of shape [bw_num_units, aux_input_size].

     * * 48: The activation function.

     *       A value indicating the activation function:

     *       <ul>
@@ -2607,8 +2662,8 @@ enum OperationType : int32_t { 
     * * “activation” is the function passed as the “fused_activation_function”

     *   argument (if not “NONE”).

     *

     * The op also supports an auxiliary input. Regular cell feeds one input

     * into the two RNN cells in the following way:

     * The op supports cross-linking via an auxiliary input. Regular cell feeds

     * one input into the two RNN cells in the following way:

     *

     *       INPUT  (INPUT_REVERSED)

     *         |         |
@@ -2618,8 +2673,8 @@ enum OperationType : int32_t { 
     *         |         |

     *      FW_OUT     BW_OUT

     *

     * An op with an auxiliary input takes two inputs and feeds them into the

     * RNN cells in the following way:

     * An op with cross-linking takes two inputs and feeds them into the RNN

     * cells in the following way:

     *

     *       AUX_INPUT   (AUX_INPUT_REVERSED)

     *           |             |
@@ -2632,9 +2687,26 @@ enum OperationType : int32_t { 
     *         |           |

     *      FW_OUT      BW_OUT

     *

     * The cross-linking mode is enabled iff auxiliary input and auxiliary

     * weights are present. While stacking this op on top of itself, this

     * allows to connect both forward and backward outputs from previous cell

     * to the next cell's input.

     *

     * Since HAL version 1.3 parallel linking mode is supported. The mode is

     * enabled if auxiliary input is present but auxiliary weights are omitted.

     * In this case, the cell feeds inputs into the RNN in the following way:

     *

     *       INPUT (AUX_INPUT_REVERSED)

     *         |         |

     *    ---------------------

     *    | FW_RNN     BW_RNN |

     *    ---------------------

     *         |         |

     *      FW_OUT     BW_OUT

     *

     * While stacking this op on top of itself, this allows to connect both

     * forward and backward outputs from previous cell to the next cell's

     * inputs.

     * corresponding inputs.

     *

     * Supported tensor {@link OperandType}:

     * * {@link OperandType::TENSOR_FLOAT16}
@@ -2667,11 +2739,17 @@ enum OperationType : int32_t { 
     *      A 2-D tensor of shape [batchSize, bwNumUnits]. Specifies a hidden

     *      state input for the first time step of the computation.

     * * 9: auxInput.

     *      A 3-D tensor. The shape is the same as of the input 0.

     *      A 3-D tensor. The shape is defined by the input 6 (timeMajor). If

     *      it is set to true, then the input has a shape [maxTime, batchSize,

     *      auxInputSize], otherwise the input has a shape [batchSize, maxTime,

     *      auxInputSize]. Can be omitted. See the docs above for the usage

     *      modes explanation.

     * * 10:fwAuxWeights.

     *      A 2-D tensor of shape [fwNumUnits, inputSize].

     *      A 2-D tensor of shape [fwNumUnits, auxInputSize]. Can be omitted.

     *      See the docs above for the usage modes explanation.

     * * 11:bwAuxWeights.

     *      A 2-D tensor of shape [bwNumUnits, inputSize].

     *      A 2-D tensor of shape [bwNumUnits, auxInputSize]. Can be omitted.

     *      See the docs above for the usage modes explanation.

     * * 12:fusedActivationFunction.

     *      A {@link FusedActivationFunc} value indicating the activation function. If

     *      “NONE” is specified then it results in a linear activation.